How to show that a model is not over-fitted?

This might have a very simple answer, but I am doing an analysis of a financial series and have decided to use regression in order to predict a particular revenue given a set of input variables. Since this will ultimately be reported to a budget committee, we have aggregated all of the data to a monthly level so that we can use the model to make predictions of monthly totals.

Since we are working with monthly totals, we have far fewer observations than I would like (~11,000 records aggregates down to just 41 monthly totals); fitting my model with 19 predictors gives:

F(19, 21) = 11.21, p < 0.000, r2 = 0.910

However, when I presented this to my supervisor, he said that the good fit was probably spurious because the model contains so many predictors and so few free observations. Out of curiosity, I fit the model to the unaggregated data (individual records), getting:

F(19, 1227) = 268.3, p < 0.000, r2 = 0.806

On the basis of r2 alone, I'm convinced that overfitting does not seem to be the cause of the good fit at the monthly level (otherwise why would it fit a much larger dataset with only a small penalty?).

So, are there common tests I could perform or statistics to present that would help me convince myself (or my supervisor) that the monthly model is not overfitted?

• 19 predictors with 41 observations is almost certainly resulting in some overfitting, the main question is how bad it is - did you also perform some variable selection from even more predictors? Jul 6 '16 at 17:26
• Yes, the question here is, "given a model that performs fairly well, how can we show that its performance is due to a 'true fit' as opposed to an artifact of overfitting?" I'm interested in your approach - how would one use variable selection procedures to defend this?
– Gabe
Jul 6 '16 at 18:50
• My question on variable selection was about whether the situation is even worse than the number of variables and records suggests. Ideal would be to assess how well the model predicts more (completely independent) data. Jul 6 '16 at 19:56
• I don't believe you should be thinking of overfitting as something that exists or doesn't. Virtually all models are going to be overfit to the data that were used in model-building, so the point is to use independent data to measure performance. Your approach makes some sense, but it would be better to use data that the model has never seen before (your model has seen these data in the form of averages). Jul 6 '16 at 21:14

If you like using $R^2$, then I would recommend using adjusted $R^2$ (denoted $R^2_{adj}$) instead of $R^2$. $R^2_{adj}$ penalizes for overfitting in the sense that a higher value is still better, but adding variables that contribute very little may cause $R^2_{adj}$ to decrease. Calculating $R^2_{adj}$ for multiple models will allow you to select a model that balances variance explained by your predictors and overfitting. While you may use this for a small number of models, it is inadvisable to fit many models and select the model with the highest $R^2_{adj}$ value. (I would personally recommend building two or three models with which you are comfortable and comparing the $R^2_{adj}$ values, but no more.)
I may also recommend using Mallow's $C_p$, which is a statistic easily generated in many computer softwares. Mallow's $C_p$ can be calculated for multiple models (much like $R^2$ or $R^2_{adj}$) and you would want to select the model where $C_p$ comes closest to the number of predictors $p$ in that particular model. (For example, if model 1 had four predictors and model 2 had seven, you would select model 1 if $C_{p,1}$ was closer to 4 than $C_{p,2}$ was to 7, where $C_{p,i}$ is the value of Mallow's $C_p$ for model $i$.)
• if you calculate adjusted r square for the example $1 - (1-0.910)\times\frac{41-1}{41-19-1}=0.828$. This is very close to the R square from the unit level model. probably a coincidence/quirk but I found it interesting. Jun 15 '20 at 13:33