What statistics should I use for evaluating the accuracy of predictions? I have two variables representing 1) players' predicted fantasy football points and 2) players' actual fantasy football points scored.  What statistics are best for assessing the accuracy of the predictions in terms of absolute agreement (not just relative rank-order accuracy) in relation to the actual values?  I'd like to compare the accuracy of various sources of projections.  The predicted and actual values are both continuous.  I'm considering statistics like: r-squared, Harrell's c-index, and ICC (absolute agreement).  Which statistic(s) would you recommend and why?
I'd like to use R (and any necessary packages) to compute the prediction accuracy.
Thanks in advance!
 A: Ask yourself first "why am I making predictions?". If you are trying to predict something, it usually means you (or someone you advise) are intending to take some action based on that prediction. Therefore you need to ask yourself what is most important for your application? Different measures of accuracy are derived using different assumptions about the data, models and the utility of the predictions. In order to determine which one is best for your situation, you first need to think about these questions as they apply to you, then carefully look into the rationale behind different measures of accuracy to see which might apply.
If you re-formulate your question to give people more information, that might help to get some specific suggestions.
A: You're going down the wrong track talking about "agreement". Things like ICC have to do with raters, not predictors. I'd recommend a paper by frequent-contributor Rob Hyndman to you.
I've edited this since my previous answer referred to MAD but non-thinkingly pointed you to the wikipedia page referring to MAD as a measure of spread in a univariate series. What I meant was MAD as in Mean Absolute Deviation, which is more clearly known as MAE (Mean Absolute Error). Cursed acronyms!
Are you doing this by hand, or using Excel, R, or some kind of package to do it?
