How to know if a statistic prediction is right?

Question

The weather forecast predict the probability of rain or not for some day. If I could repeat the same day many times I could count how many times does it rain or not, so I could compare with the prediction to know if the prediction was right.

The fact is that is not possible to repeat a day, the only data that I have is the corresponding predictions for many different days and only one point in distribution for each prediction.

So how could I know if the probabilities for the weather forecast are correct?

This question can be extended to any prediction in which I can not repeat the phenomena many times to obtain an entire distribution and verify the prediction.

Answer 1

"This question can be extended ..." -- that's absolutely right. But of course, if you want to step all the way back -- that's the case for every phenomenon. Every time you flip a coin, it gets a little dented, and changes the likelihood of coming up heads. Every time you shoot a basket, your arms are a little more tired (or a little better rested) and your chance of the ball going in are just a little different.

As an applied statistician, an enormous part of your job is trying to determine what events are similar enough to be counted as the same. You will never have a bunch of people taking drugs, or a bunch of students being tested, or a bunch of cities implementing policies, that are exactly the same. Much of the meat of your job is in trying to determine what to control for so that, when you're done, they're similar enough to give you back a meaningful answer.

When it comes to predictions, the best you can do is try to train, and then test, on things you think are sufficiently similar. The whole point of cross-validation is to examine how internally consistent your data and model are. If you can train on some, and accurately predict on the rest, a solid interpretation is that the two sets of data are "similar enough." (Assuming away the other enormous part, that your model is correct.) So for observed data, you can assess predictive accuracy with cross-validation.

But for the unseen future, the best answer to your question is just "For the predictions to be correct, you have to assume that tomorrow's weather is drawn from the same distribution as all the weather on which the predictive model was fit." And any question of how close becomes dependent upon a particular model, and preference.

Answer 2

This is a great question and a common one as well. The property you seem to be interested in is ergodicity. If a stochastic process you're interested in is ergodic, then (roughly) these "different day" observations that you see can be combined to assess how successful the weather predictions are; can be combined to derive some convergence results. If the process does not exhibit ergodicity, however, then—as you stated—one would need to observe the same day several times and see if this probability of rain is accurate or not. Ergodicity is difficult to verify with real data and is typically taken as an assumption.

For a rigorous, but empirical treatment of ergodicity, check out this chapter of E. Zivot's time series book. For a very nice intuitive example, watch this video from 16:55.

Answer 3

Time series econometrics deals with a similar question: If $y_t$ and $x_t$ are time series variables, should you trust a linear regression with the two variables? The answer is "it depends".

It depends on whether the observed relationship between the two variables would continue to be true in the future. If $y_t$ and $x_t$ are both non-stationary, then the observed relationship may break apart in the future. If $y_t$ and $x_t$ are both stationary, then the observed relationship should hold in the future.

Here's a simulated example. The variables, $x_t$ and $y_t$, are both non-stationary by design. Although the regression model says that the observed relationship is strong (based on p-value and $R^2$), the out-of-time $R^2$ is horrible (the model is far worse than using the average as a prediction).

### create two non-stationary variables
set.seed(12345)

x <- 100 + cumsum(rnorm(1000))

y <- 200 + cumsum(rnorm(1000))

df <- data.frame(y=y, x=x)

### split between training and test

train <- df[1:800, ]  ## 80% train
test <- df[801:1000, ] ## 20% train

### linear regression

lm.mod <- lm(y~x, data=train)

summary(lm.mod)

### measure fit

library(caret)

in.sample.R2 <- R2(lm.mod$fitted.values, train$y, formula="traditional")
out.sample.R2 <- R2(predict(lm.mod, newdata=test), test$y, formula="traditional")

in.sample.R2
out.sample.R2

TLDR; Predicting the future is hard. Linear regression using time series data may be extremely misleading. Hold-out some of your data based on sequential time (e.g., hold out the last 9 quarters of your time series). Validate your model using the hold-out data.