1
$\begingroup$

Say I am performing a hypothetical measurement study of the time taken by different computers to finish a specific task eg. download a list of web pages $w \in W$ and store the results of each page load in an array $T$. Using the results obtained so far, I make a prediction that by changing some parameter like the file chunk size the download times can be reduced to $P$ which is a list of floating point values corresponding to the expected download time. At a later time, by actually performing an experiment to validate the hypothesis of file chunk size improving download time I obtain the actual times $A$.

Is a t-Test (scipy.stats.ttest_ind(P, A, equal_var=True)) the correct way to check and conclude that the predictions made in $P$ were accurate and reliable? Are there other statistical tests which I could use here to indicate the accuracy of the predictions compared to the experimental results?

$\endgroup$
1
$\begingroup$

There are many measures of forecast accuracy. (That online forecasting textbook is concerned with time series forecasting, but the KPIs can be applied in any prediction.) We also have tags for the , , or . You may be interested in a short paper of mine (Kolassa, 2020, IJF) in making sense of how these various measures hang together.

You can in principle use a t-test to assess whether your predictions have the same mean as your actuals. One question is whether this is really interesting. If you have $n$ predictions, and they are on average too high by $x$, then you can subtract $nx$ from any one prediction, and the resulting vector of predictions will have the same mean as your actuals. Does this mean that this ad hoc modified prediction is better than the original one? I wouldn't think so.

In the time series forecasting community, this kind of testing whether predictions have the same mean as actuals is usually not done, likely because of problems like this. Rather, the focus is on whether predictions are "good enough", or whether a particular prediction/forecasting algorithm improves on a very simple benchmark method, like the overall mean (which can be surprisingly hard to beat).

(Also, if you do decide to use a t-test, don't use equal_var=True. Your predictions will almost certainly have a lower variance than the actuals, simply because in predicting, you filter out unpredictable noise.)

$\endgroup$
1
  • $\begingroup$ Thank you so much. This is very helpful. $\endgroup$ Jan 28 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.