predictive modeling: comparing actual and predicted values in terms of accuracy I have applied a predictive model on a hold out data set on which I know actual values of the target variable. I wonder how to compare actual and predicted forecasted values, verifying whether on average the two values are statistically the same. I am not assessing the precision of estimate, but the accuracy: I want to assess whether on average $\hat{Y_i}=Y_i$. I am not bothering about the precision (i.e. RMSE or $R^2$) but in non - distortion (i.e. $E \left[ \hat{Y_i} \right]$=$Y_i$).  I have used paired sample t.test but I am not sure whether this is the right approach to assess accuracy.
 A: Does it make sense to compute R^2? It won't give you a "yes they are the same" or "no they are different" answer, but it will tell you how much of the variability in Yi is accounted for by your predictions.
As for your paired-samples t.test approach, there are two potential, related issues. First, it seems you would like to conclude that the predicted and actual values are not different.
This is problematic with significance testing because one's ability to detect differences is a function of sample size - the larger your sample size, the more likely it is that any difference between the two values will be significant.
However, if your sample size is small, you will not be able to detect all but the largest of differences between your actual and predicted values, and thus, may erroneously reach the conclusion that the predicted and actual values are "not different".
All in all, this approach would seem quite wishy-washy (subjective; how do you determine whether your sample size is large "enough"?), and not provide any solid answers.
A: One of the appropriate ways of evaluating regression models is Mean Squared Error (commonly referred to as MSE).
$$MSE = \frac1n \sum (\hat{Y}_i - Y_i )^2$$
If you want your evaluation metric to be in the same units as the data you're working with, you may prefer to use Root Mean Square Error (RMSE).
$$RMSE = \sqrt{(\frac1n \sum (\hat{Y}_i - Y_i})^2$$
You'll want to try a couple of different regression methods, and generally you'll take the one with the smallest MSE (or reject all of them and look for a better solution).
