I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). Specifically, I'm curious whether the final accuracy measure used (e.g. explained variance) should be an average of such measure for individual cross sections (perhaps to accounting for distribution of the accuracy measure itself) or of the full data sample?
To give an example, when dealing with factor models in quantitative finance, you have cross sections of variable number of stocks with multiple attributes, which you use to forecast the return of the stock for some lagged period.
To evaluate the in- and out- sample performance, I have seen both methods applied, for example:
- IC Score (cross-sectional correlation averaged over the sample period)
- R2/Explained Variance model on the full sample
For fixed-effect models (e.g. Linear Regression), I understand the logic behind using the full sample, but I also find that it masks the presence of shocks/reversals in the system that may be occurring in specific periods that is unaccounted for in the model?
Also, when it come to tuning parameters for learning algos, should I adjust generic evaluations when it comes to cross-validation so that cross-sections are evaluated together?