# Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a given month, so it is zero for ~80% of the people in any month, but there's a long right tail of heavy users. The predictions I have do not seem to respect the count nature of the outcome: fractional concerts are prevalent.

I don't know anything about the models. I only observe 6 different black-box predictions $\hat y_1,...,\hat y_6$ for each person per month. I do have an extra year of data that the model builders did not have for the estimation (though the concert-goers remain the same), and I would like to gauge where each performs well (in terms of accuracy and precision). For instance, does some model predict well for frequent concert-goers, but fail for the couch potatoes? Is the prediction for January better than the prediction for December? Alternatively, it would be nice to know that the predictions allow me to rank people correctly in terms of the actuals, even if the exact magnitude cannot be trusted.

My first thought was to run a fixed effects regressions of actual on predicted and time dummies and look at the RMSEs or $R^2$ for each model. But that does not answer the question about where each model does well or if the differences are significant (unless I bootstrap the RMSE). The distribution of the outcome also worries me with this approach.

My second idea was to bin the outcome into 0, 1-3, and 3+, and calculate the confusion matrix, but this ignores the time dimension, unless I make 12 of these. It's also pretty coarse.

I am aware of Stata commands concord by T.J. Steichen and N.J. Cox--which has the by() option, but that would require collapsing the data to annual totals. This calculates Lin's Concordance Correlation Index with confidence intervals, among other useful stats. CCC ranges from -1 to 1, with perfect agreement at 1.

There's also Harrell's $c$ (calculated through somersd by R. Newson), which has the cluster option, but I am not sure that would allow me to deal with the panel data. This gives you confidence intervals. Harrell's c is the generalization of the area under a ROC curve (AUC) for a continuous outcome. It's the proportion of all pairs that can be ordered such that the subject with the higher prediction actually has the higher outcome. So $c=0.5$ for random predictions $c=1$ for a perfectly discriminating model. See Harrell's book, p.493

How would you tackle this problem? Would you suggest calculating statistics like MAPE that are common in forecasting?

Useful things found so far:

• Slides on a repeated measure version of Lin's Concordance Correlation Coefficient
• We'll need to know more about the behavior, is it ordinal/binary/continuous valued? Because this experiment is longitudinal, does your interest lie in forecasting or predicting outcomes in an individual? Mixed effects models are used for inference, not prediction. They don't work because, in order to predict, you need an estimate of the random effect. Commented Jun 6, 2013 at 23:54
• Actual behavior is either counts or continuous. The predictions are all continuous. I would like to see how good the individual level monthly predictions are. Commented Jun 6, 2013 at 23:57
• "Individual level monthly predictions" in persons whom you've observed previous data or persons at initial assessment? I.E. are you obtaining $\widehat{Y_{i=12}} = f(X_{i=12, 11, \cdots, 1}, Y_{i=11, 10, \cdots, 1}$ or $\widehat{Y_{i=I}} = f(X_{i=I, I-1, \cdots, 1}, Y_{i=I-1, I-2, \cdots, 1}$ or $\widehat{Y_{i}} = f(X_{i})$ or ... please clarify as prediction from longitudinal is not a hand-wavy matter. Commented Jun 7, 2013 at 0:06
• This is a sample of people who were used for the estimation, but the prediction is for the year after the estimation window. The prediction for the first month is a function of lagged, observed behavior and observed covariates for person $i$: $\hat Y_{i,1}=f(Y_{i,t-1},X_{i,t}).$ I expect the first month to be pretty close. For the second month (and thereafter), it's a function of last month's prediction and that month's observed covariates: $\hat Y_{i,2}=f(\hat Y_{i,1},X_{i,2})$. I expect this prediction to diverge from the actuals more and more. I want to compare $Y_{i,t}$ and $\hat Y_{i,t}$. Commented Jun 7, 2013 at 0:22
• estimation implies parameter estimation which may be a part of "training" for a predictive model, but I think you mean to say that your sample is used for training a predictive model. What you're stating here is a conditional semi-markov process and has unique applications in forecasting. Commented Jun 7, 2013 at 20:40

To evaluate a semi-Markov forecast's predictive ability, there are a number of methods available depending on sample size and other available information.

For evaluating any predictive/forecast model, you have the option of cross validation (specifically leave-one-out or iterative split sample cross validation), where a model is estimated in a "training" sample and model uncertainty assessed in a "validation" sample. Depending on the distribution of the outcome, a number of measures are available by which you can select a model among a panel of eligible models. For general non-parametric measures for model selection, people really like AIC and BIC, especially the latter.

CCC and c-statistics are used to evaluate binary cross-sectional predictions like from tests/assays, so you'll have to rule them out if you're predicting, say, BMI or IQ. They measure calibration (like the Hosmer Lemeshow test) and what's called risk stratification capacity. No intuitive connection to continuous outcomes there, at least not as far as I can tell.

RMSE on the other hand is used to evaluate continuous predictions (save the case of risk prediction in which RMSE is referred to as a Brier score, a pretty archaic and deprecated model evaluation tool). This is an excellent tool and probably is used to calibrate upwards of 80% of predictive models we encounter daily (weather forecasts, energy ratings, MPG on vehicles, etc.).

A caveat in split sample validation or resampling for evaluating forecast models is that you may only be interested in future outcomes when your sample leaves you predicting past outcomes. Don't do this! It doesn't reflect the models application and can vastly sway selection in a negative fashion. Roll forward all available information and predict future, unobserved outcomes in all available cases.

Pretty much any applied linear models book will cover prediction, RMSE, and the nuances of training and validating models. A good start would be Kutner, Nachtsheim, Neter, Li, also consider Diggle's "Time Series Analysis", Diggle Heagerty Zeger Li's, "Longitudinal Data Analysis", and potentially Harrell's "Regression Modeling Strategies".

• CCC and Harrell's c can be used with continuous outcomes. CCC also has a repeated measures implementation. See the references/links I added in the question. Commented Jun 10, 2013 at 18:02
• Doesn't matter. You're not classifying. Commented Jun 10, 2013 at 18:15