The weather forecast predict the probability of rain or not for some day. If I could repeat the same day many times I could count how many times does it rain or not, so I could compare with the prediction to know if the prediction was right.

The fact is that is not possible to repeat a day, the only data that I have is the corresponding predictions for many different days and only one point in distribution for each prediction.

So how could I know if the probabilities for the weather forecast are correct?

This question can be extended to any prediction in which I can not repeat the phenomena many times to obtain an entire distribution and verify the prediction.

  • 1
    $\begingroup$ If predictions are probabilistic (e.g. each prediction is a percentage chance of rain) you can compare the percentages in bands with the proportion of times it did rain for those, like a p-p plot. Actual forecasts tent to be rounded to multiples of 5% or 10% in any case so the bands are often already there. $\endgroup$ – Glen_b Dec 31 '15 at 3:43
  • $\begingroup$ It seems a method that works, I should have a lot of data to have repeated predictions (same percentages) for different days, than I could count how many days rain or not and compare with that prediction. But is not clear how I could know visually if the forecast is more right or wrong over the predictions. i would like a numerical value indicating if and how much is right or wrong. $\endgroup$ – Rafael Dec 31 '15 at 4:18
  • $\begingroup$ Did you look at ROC (Receiver Operating Characterstic) , Brier score to compare the relative performance of two predictions ? $\endgroup$ – user83346 Jan 2 '16 at 11:43
  • $\begingroup$ Not yet, but I will. $\endgroup$ – Rafael Jan 4 '16 at 1:36

"This question can be extended ..." -- that's absolutely right. But of course, if you want to step all the way back -- that's the case for every phenomenon. Every time you flip a coin, it gets a little dented, and changes the likelihood of coming up heads. Every time you shoot a basket, your arms are a little more tired (or a little better rested) and your chance of the ball going in are just a little different.

As an applied statistician, an enormous part of your job is trying to determine what events are similar enough to be counted as the same. You will never have a bunch of people taking drugs, or a bunch of students being tested, or a bunch of cities implementing policies, that are exactly the same. Much of the meat of your job is in trying to determine what to control for so that, when you're done, they're similar enough to give you back a meaningful answer.

When it comes to predictions, the best you can do is try to train, and then test, on things you think are sufficiently similar. The whole point of cross-validation is to examine how internally consistent your data and model are. If you can train on some, and accurately predict on the rest, a solid interpretation is that the two sets of data are "similar enough." (Assuming away the other enormous part, that your model is correct.) So for observed data, you can assess predictive accuracy with cross-validation.

But for the unseen future, the best answer to your question is just "For the predictions to be correct, you have to assume that tomorrow's weather is drawn from the same distribution as all the weather on which the predictive model was fit." And any question of how close becomes dependent upon a particular model, and preference.

| cite | improve this answer | |
  • $\begingroup$ I'm not as strict, an approximate answer to some degree is enough to represent reality. And I do not make predictions, I would have more knowledge for this, just want to test the predictions of others. $\endgroup$ – Rafael Dec 31 '15 at 4:34
  • 1
    $\begingroup$ But statistics is all about degrees -- how close do you need to get? Choosing that degree isn't a simple problem. $\endgroup$ – one_observation Dec 31 '15 at 4:37
  • 1
    $\begingroup$ Methods like cross-validation (eg w/ the Brier score) can be used to get the out of sample predictive accuracy. $\endgroup$ – gung - Reinstate Monica Dec 31 '15 at 4:39
  • $\begingroup$ Close to the point where I do not see. $\endgroup$ – Rafael Dec 31 '15 at 4:39
  • $\begingroup$ That blind, is the site name and I did not even see. $\endgroup$ – Rafael Dec 31 '15 at 4:47

This is a great question and a common one as well. The property you seem to be interested in is ergodicity. If a stochastic process you're interested in is ergodic, then (roughly) these "different day" observations that you see can be combined to assess how successful the weather predictions are; can be combined to derive some convergence results. If the process does not exhibit ergodicity, however, then—as you stated—one would need to observe the same day several times and see if this probability of rain is accurate or not. Ergodicity is difficult to verify with real data and is typically taken as an assumption.

For a rigorous, but empirical treatment of ergodicity, check out this chapter of E. Zivot's time series book. For a very nice intuitive example, watch this video from 16:55.

| cite | improve this answer | |
  • $\begingroup$ I had no idea ergodicity, interesting to know, in my example I assume that is possible to verify predictions both repeating the event many time and with many events which can be repeated only one time, so I assume ergodicity (I think). But I dont know how to verify with ergodicity if the prediction is right. $\endgroup$ – Rafael Dec 31 '15 at 4:27

Time series econometrics deals with a similar question: If $y_t$ and $x_t$ are time series variables, should you trust a linear regression with the two variables? The answer is "it depends".

It depends on whether the observed relationship between the two variables would continue to be true in the future. If $y_t$ and $x_t$ are both non-stationary, then the observed relationship may break apart in the future. If $y_t$ and $x_t$ are both stationary, then the observed relationship should hold in the future.

Here's a simulated example. The variables, $x_t$ and $y_t$, are both non-stationary by design. Although the regression model says that the observed relationship is strong (based on p-value and $R^2$), the out-of-time $R^2$ is horrible (the model is far worse than using the average as a prediction).

### create two non-stationary variables

x <- 100 + cumsum(rnorm(1000))

y <- 200 + cumsum(rnorm(1000))

df <- data.frame(y=y, x=x)

### split between training and test

train <- df[1:800, ]  ## 80% train
test <- df[801:1000, ] ## 20% train

### linear regression

lm.mod <- lm(y~x, data=train)


### measure fit


in.sample.R2 <- R2(lm.mod$fitted.values, train$y, formula="traditional")
out.sample.R2 <- R2(predict(lm.mod, newdata=test), test$y, formula="traditional")


TLDR; Predicting the future is hard. Linear regression using time series data may be extremely misleading. Hold-out some of your data based on sequential time (e.g., hold out the last 9 quarters of your time series). Validate your model using the hold-out data.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.