# Evaluating the quality of a prediction when performing a measurement study

Say I am performing a hypothetical measurement study of the time taken by different computers to finish a specific task eg. download a list of web pages $$w \in W$$ and store the results of each page load in an array $$T$$. Using the results obtained so far, I make a prediction that by changing some parameter like the file chunk size the download times can be reduced to $$P$$ which is a list of floating point values corresponding to the expected download time. At a later time, by actually performing an experiment to validate the hypothesis of file chunk size improving download time I obtain the actual times $$A$$.

Is a t-Test (scipy.stats.ttest_ind(P, A, equal_var=True)) the correct way to check and conclude that the predictions made in $$P$$ were accurate and reliable? Are there other statistical tests which I could use here to indicate the accuracy of the predictions compared to the experimental results?

You can in principle use a t-test to assess whether your predictions have the same mean as your actuals. One question is whether this is really interesting. If you have $$n$$ predictions, and they are on average too high by $$x$$, then you can subtract $$nx$$ from any one prediction, and the resulting vector of predictions will have the same mean as your actuals. Does this mean that this ad hoc modified prediction is better than the original one? I wouldn't think so.
(Also, if you do decide to use a t-test, don't use equal_var=True. Your predictions will almost certainly have a lower variance than the actuals, simply because in predicting, you filter out unpredictable noise.)