Overcoming 70 years of insignificant rainfall enhancement trial results

The experimental design and the statistical methodology deployed in the analysis of Australian Rain Technologies’(ART) trials of the Atlant technology in the Hajar Mountains of Oman were developed in our early trials of the technology in Queensland in 2007 and in South Australia over 2008-2010. They have been fine-tuned since with an overall consistency in approach since the first South Australian trial in 2008.

This work has been done by independent econometricians and applied statisticians at Analytecon and the National Institute for Applied Statistics Research Australia (NIASRA) at the University of Wollongong.

This has allowed unprecedentedly high statistical significance in rainfall enhancement trial results. Whereas 70 years of chemical cloud seeding trials have yielded no statistically significant results, ART’s trials in Oman have shown results with statistical probabilities in the high 99 per cents. This paper sets out how such significance was achieved.

Summarised difference in approach

Chemical cloud seeding trials have worked with average rain gauge data alone, and consequently small effective sample sizes, with corresponding high standard errors and low statistical confidence.

The NIASRA/Analytecon approach uses all relevant data across all trial days and across all gauge locations and applies statistical techniques to maximize the signal/noise ratio while simultaneously dealing with the correlations in the daily rain gauge data. This has resulted in much reduced standard errors and increased statistical confidence.

Chemical cloud seeding statistical methodologies – Under-utilized data

Chemical cloud seeding trials since the 1950s have deployed a random cross-over design, simply comparing rainfall measured by gauges in target and control areas, without consideration of other variables such as meteorological conditions. Rain gauge measurements in target and control areas have been averaged across space and time to deal with correlation between these measurements. This has led to seriously reduced effective sample size, greatly increased standard errors and thus low statistical probabilities that enhancement had actually occurred.

Steps to statistical significance - The NIASRA/Analytecon approach

The methods deployed to arrive at the lower standard errors associated with enhancement estimation in the Oman trials include:

Methods to minimize standard error - (through more robust modelling)

• Use of daily rainfall data from an extensive set of rain gauges (without averaging) and daily meteorological data. Exhaustive modeling of meteorological and topographic covariates to increase signal-to-noise ratio. The modelling across all gauges and days despite correlation between them is addressed below.

• Use of a downwind rainfall model to make counterfactual predictions of rainfall without any enhancement effect. The natural downwind rainfall is estimated via an instrumental variable based on observed upwind rainfall (which is by definition unconnected with operation of the technology).

• Using analysis methods that minimize the impact of the extreme variability in rainfall by modeling the likelihood of rainfall on any particular day separately from modeling the amount of rainfall recorded on days when there is rain. This eliminates much of the “noise” in the rainfall data due to its episodic nature, and enables effective isolation of any actual rainfall enhancement.

• Modeling actual rainfall on a scale (the logarithmic) that is robust to outlying values of observed rainfall.

• Minimizing bias in the rainfall analysis by using models that are robust to non-inclusion of unobserved causal variables, thus avoiding incorrectly assigned effects. This includes the use of day and gauge-level random effects to allow for missing temporal and spatial factors impacting on downwind rainfall.

Block bootstrapping - allowing for correlation without ballooning standard errors

• Robust spatiotemporal resampling via a two-level nonparametric block bootstrap to avoid overstating confidence in the estimated enhancement effect by understating its standard error (i.e. by including all data points from correlated gauges without allowing for that correlation). This methodology allows us to create a distribution of potential attribution values from which valid confidence intervals and associated significance levels can be extracted.

Permutation testing - a further check on actual rainfall enhancement

• Use of permutation testing as a further check on whether the estimated attribution value is clearly a consequence of the actual operation of the enhancement technology. This is done by scrambling the daily operating schedules of the Atlant mechanisms a large number of times in order to create a large number of potential alternative operating schedules, and hence a large number of estimated enhancement effects corresponding to these scrambled operating schedules. This distribution of potential enhancement effects is then compared with the actual estimated enhancement effect based on the actual operating schedule.

If there is no true enhancement, then the operating schedule should have no impact on any calculated enhancement, which then could be due to specific meteorological factors not allowed for in the analysis. In this case, the actual estimated enhancement given the true operating schedule should lie within the main part of this "permutation distribution" of alternative enhancement effects. On the other hand, if in fact there is an effect specifically tied to the actual operating schedule then we should see a value for the actual estimated enhancement that lies at or near the positive extreme of this permutation distribution.

The figure below shows the Permutation Test attributions with the highest attribution being for the actual schedule permutation (18.8 per cent).

The permutation distribution of estimated attribution for the combined five Atlant field trials