Independence of samples

Question

Recently I started doing verification on daily temperature forecasts by different providers. I calculated the absolute forecast errors for each of the providers, and I am planning on performing ANOVA to check if the differences in mean absolute error between the providers is significant. So lets say that my sample consists of 110 maximum daily temperature forecast errors for 5 different providers. The forecasts errors are obtained as the difference between the actual temperature, obtained from the nearest weather reporting station and the forecasts by each of the providers.

What puzzles me here is the independence of the sample, since I use same set of actual temperature values to obtain the errors for each of the forecasts. The more I read about independence of samples the more I get confused. Is the independence assumption violated in this case?

Moreover, the distribution of the sample is nearly exponential, so I tried with transformation to approximate it to normal distribution. However, the normality significance tests indicate that the distribution is still non-normal, with a p-value of 0.01. How can I determine if the normality distribution assumption is not seriously violated, since I read on many threads that the test can be performed even if the normality assumption is moderately violated?

Answer 1

The assumptions of tests like ANOVA need to be assessed using knowledge about the science behind the data and how it was collected.

In your case I would expect that a one-way ANOVA is probably not the best approach due to non-independence. Think about if the forecasters are working from essentially the same information and there is an unexpectedly hot or cold day, all the forecasters are likely to be off by similar amounts for that day, so the values would not be independent.

The real requirement is conditional independence, meaning that if you can condition on what causes the dependence (the actual temperature) and assume independence after the conditioning, then the model will be fine. You could look into a randomized block design (with the date/observed temperature forming the blocks) or a mixed effects model (not for beginners, consult with a local statistician).

You might also want to think about what question you are trying to answer. What if one forecaster is usually 5 degrees to low, another forecaster is usually 5 degrees too high, and a third is usually off by 5 degrees, but sometimes too high and sometimes too low. All 3 have about the same average absolute difference, but are they really the same? This is a case where a difference in variances may be as interesting or more interesting than the difference in means. Figuring out which differences would be the most interesting, then finding the test to look for those differences is better than just figuring out if the assumptions of a common test hold.