Short version

When averaging over a presumed stationary time series and calculating statistics (e. g. normalized mean square error) to compare to a simulation (atmospheric turbulence model) of the same time series, how much does the power decrease?

Normally statistical power scales with $\sqrt{N}$, but I don't think that is true if I average over, say, six data points, leading to $N/6$ many. I need to know whether one "good" value is better than six "bad" ones.

When I choose thirty minute averaged data, I have six simulation/observation pairs per day (each containing 14 data points). When I choose three hour averaged data, I have only one simulation/observation pairs per day (each containing 14 data points).

Long version

I have a computer model that simulates air pollution concentration in a stationary situation. Stationary in my field means that all time derivatives are zero. I already found out that this is weaker than a "stationary time series" in statistics. Basically the model assumes that the wind direction, wind speed and some turbulence parameters (which are input at the start) do not change during the simulated time.

To validate the model I have four days of field measurements where a tracer substance was released at a source with a constant flow and measured at fourteen different positions for three hours. The days of the experiments were chosen to have as stationary conditions as possible and I have already excluded one day because the wind direction monotonously changed during the experiment, which the model cannot handle. On another day the wind direction oscillated relatively heavily around one direction. Therefore short averaging periods pose a problem, because the wind blows the tracer in different directions and the necessary steady state is not achieved. For longer averaging periods this seems to even out. I have an estimation for the required averaging period to exclude turbulence with $\alpha$ significance to support this. I chose to omit it here, but can easily supply it, if necessary.

For technical reasons I can only have the measured input and output variables with thirty minute, one hour or three hour averaging periods (they are measurements of turbulent variables).

The model predicts concentration at the measurement positions and since the model is always stationary and the measurements are assumed to be (roughly) stationary, the values should ideally be identical, but at least comparable.

This comparison is undertaken via several statistical indices like the normalized mean square error (NMSE), fractional bias, etc. I bootstrap these calculations with a fixed block (one day) and use the Studentized method to calculate confidence intervals. I know that this is not perfect, but the focus of my thesis is on the model, not the statistics.

Now I need to compare different model versions with each other to see whether a change improved the model or not. Following Hanna (CONFIDENCE LIMITS FOR AIR QUALITY MODEL EVALUATIONS, AS ESTIMATED BY BOOTSTRAP AND JACKKNIFE RESAMPLING METHODS, 1989, Atmospheric Environment Vol. 23) I calculate the difference of the means of the bootstrap sample statistic between two model versions and check whether the confidence interval contains zero. For example, for the NMSE between model A and B: draw a sample with replacement, calculate $\text{NMSE}_A-\text{NMSE}_B$ and redo that 50 000 times. The result is a distribution of differences, from where I calculate the confidence interval as described above.

Now my problem is: When I choose a longer averaging period, I have less data points. However, the agreement of the model and the measurements becomes better and the variance of the bootstrapped statistical index also becomes smaller (less "bad" values). Theoretically more data should lead to more statistical power (we're talking about up to a factor of six more data points). However, it can't be a "normal" six times more data, because some of the information is still contained due to the averaging.

I'm a bit lost on finding a method to choose the best averaging period to run my model and statistics on. Shorter gives more points, longer much better model performance. When I choose thirty minute averaged data, I have six simulation/observation pairs per day (each containing 14 data points). When I choose three hour averaged data, I have only one simulation/observation pairs per day (each containing 14 data points).

Disclaimer: I have hardly any knowledge about statistics.

  • $\begingroup$ Good question! I suggested some edits for readability and copied a paragraph from the long version to the shorty version; feel free to roll them back (from the edit history on the Edit page) if you don't agree with them. $\endgroup$ – shadowtalker Feb 1 '15 at 14:59
  • $\begingroup$ Also I just realized you might have made a typo in the sentences I copied. You list the same sample size for the thirty minute and three hour data (unless I broke something with my edit) $\endgroup$ – shadowtalker Feb 1 '15 at 15:01
  • $\begingroup$ And apologies for the triple post, but now a statistical comment: the bootstrap is a large sample approximation and its validity depends on your sample being an approximation to the population. I'm not sure how it works in the presence of a finite population but it might not be doing what you think it's doing. And on an unrelated point, be very, very careful about how you interpret that confidence interval. $\endgroup$ – shadowtalker Feb 1 '15 at 15:04
  • $\begingroup$ @ssdecontrol: Thank you for your help, I fixed the sample size typo. I think (and I may be wrong here) that my use of bootstrap is valid because I don't bootstrap data points, but statistics that themselves describe the correlation between data points. The number of samples is probably a problem, but they are either 3x14=42 or 2x6x14=168, so not that low. $\endgroup$ – StefanS Feb 1 '15 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.