Prediction evaluation metric for panel/longitudinal data I would like to evaluate several different models that provide
predictions of behavior at a monthly level. The data is balanced, and
$n=$100,000 and $T=$12. The outcome is attending a concert in a given
month, so it is zero for ~80% of the people in any month, but there's
a long right tail of heavy users. The predictions I have do not seem
to respect the count nature of the outcome: fractional concerts are
prevalent.
I don't know anything about the models. I only observe 6 different
black-box predictions $\hat y_1,...,\hat y_6$ for each person per month. I do
have an extra year of data that the model builders did not have for
the estimation (though the concert-goers remain the same), and I would
like to gauge where each performs well (in terms of accuracy and
precision). For instance, does some model predict well for frequent
concert-goers, but fail for the couch potatoes? Is the prediction for
January better than the prediction for December? Alternatively, it
would be nice to know that the predictions allow me to rank people
correctly in terms of the actuals, even if the exact magnitude cannot
be trusted.
My first thought was to run a fixed effects regressions of actual on
predicted and time dummies and look at the RMSEs or $R^2$ for each model. But
that does not answer the question about where each model does well or
if the differences are significant (unless I bootstrap the RMSE). The distribution of the outcome also worries me with this approach.
My second idea was to bin the outcome into 0, 1-3, and 3+, and
calculate the confusion matrix, but this ignores the time dimension,
unless I make 12 of these. It's also pretty coarse.
I am aware of Stata commands concord by T.J. Steichen and N.J. Cox--which has the by() option, but that would require collapsing the data to annual totals. This calculates Lin's Concordance Correlation Index with confidence intervals, among other useful stats. CCC ranges from -1 to 1, with perfect agreement at 1.
There's also Harrell's $c$ (calculated through
somersd by R. Newson), which has the cluster option, but I am not
sure that would allow me to deal with the panel data. This gives you confidence intervals. Harrell's c is the generalization of the area under a ROC curve (AUC) for a continuous outcome. It's the proportion of all pairs that can be ordered such that the subject with the higher prediction actually has the higher outcome. So $c=0.5$ for random predictions $c=1$ for a perfectly discriminating model. See Harrell's book, p.493
How would you tackle this problem? Would you suggest calculating statistics like MAPE that are common in forecasting?

Useful things found so far:


*

*Slides on a repeated measure version of Lin's Concordance Correlation Coefficient 

 A: To evaluate a semi-Markov forecast's predictive ability, there are a number of methods available depending on sample size and other available information. 
For evaluating any predictive/forecast model, you have the option of cross validation (specifically leave-one-out or iterative split sample cross validation), where a model is estimated in a "training" sample and model uncertainty assessed in a "validation" sample. Depending on the distribution of the outcome, a number of measures are available by which you can select a model among a panel of eligible models. For general non-parametric measures for model selection, people really like AIC and BIC, especially the latter.
CCC and c-statistics are used to evaluate binary cross-sectional predictions like from tests/assays, so you'll have to rule them out if you're predicting, say, BMI or IQ. They measure calibration (like the Hosmer Lemeshow test) and what's called risk stratification capacity. No intuitive connection to continuous outcomes there, at least not as far as I can tell.
RMSE on the other hand is used to evaluate continuous predictions (save the case of risk prediction in which RMSE is referred to as a Brier score, a pretty archaic and deprecated model evaluation tool). This is an excellent tool and probably is used to calibrate upwards of 80% of predictive models we encounter daily (weather forecasts, energy ratings, MPG on vehicles, etc.).
A caveat in split sample validation or resampling for evaluating forecast models is that you may only be interested in future outcomes when your sample leaves you predicting past outcomes. Don't do this! It doesn't reflect the models application and can vastly sway selection in a negative fashion. Roll forward all available information and predict future, unobserved outcomes in all available cases.
Pretty much any applied linear models book will cover prediction, RMSE, and the nuances of training and validating models. A good start would be Kutner, Nachtsheim, Neter, Li, also consider Diggle's "Time Series Analysis", Diggle Heagerty Zeger Li's, "Longitudinal Data Analysis", and potentially Harrell's "Regression Modeling Strategies".
