2
$\begingroup$

This might have a very simple answer, but I am doing an analysis of a financial series and have decided to use regression in order to predict a particular revenue given a set of input variables. Since this will ultimately be reported to a budget committee, we have aggregated all of the data to a monthly level so that we can use the model to make predictions of monthly totals.

Since we are working with monthly totals, we have far fewer observations than I would like (~11,000 records aggregates down to just 41 monthly totals); fitting my model with 19 predictors gives:

F(19, 21) = 11.21, p < 0.000, r2 = 0.910

However, when I presented this to my supervisor, he said that the good fit was probably spurious because the model contains so many predictors and so few free observations. Out of curiosity, I fit the model to the unaggregated data (individual records), getting:

F(19, 1227) = 268.3, p < 0.000, r2 = 0.806

On the basis of r2 alone, I'm convinced that overfitting does not seem to be the cause of the good fit at the monthly level (otherwise why would it fit a much larger dataset with only a small penalty?).

So, are there common tests I could perform or statistics to present that would help me convince myself (or my supervisor) that the monthly model is not overfitted?

$\endgroup$
  • $\begingroup$ 19 predictors with 41 observations is almost certainly resulting in some overfitting, the main question is how bad it is - did you also perform some variable selection from even more predictors? $\endgroup$ – Björn Jul 6 '16 at 17:26
  • $\begingroup$ Yes, the question here is, "given a model that performs fairly well, how can we show that its performance is due to a 'true fit' as opposed to an artifact of overfitting?" I'm interested in your approach - how would one use variable selection procedures to defend this? $\endgroup$ – Gabe Jul 6 '16 at 18:50
  • 2
    $\begingroup$ My question on variable selection was about whether the situation is even worse than the number of variables and records suggests. Ideal would be to assess how well the model predicts more (completely independent) data. $\endgroup$ – Björn Jul 6 '16 at 19:56
  • 1
    $\begingroup$ I don't believe you should be thinking of overfitting as something that exists or doesn't. Virtually all models are going to be overfit to the data that were used in model-building, so the point is to use independent data to measure performance. Your approach makes some sense, but it would be better to use data that the model has never seen before (your model has seen these data in the form of averages). $\endgroup$ – dsaxton Jul 6 '16 at 21:14
1
$\begingroup$

If you like using $R^2$, then I would recommend using adjusted $R^2$ (denoted $R^2_{adj}$) instead of $R^2$. $R^2_{adj}$ penalizes for overfitting in the sense that a higher value is still better, but adding variables that contribute very little may cause $R^2_{adj}$ to decrease. Calculating $R^2_{adj}$ for multiple models will allow you to select a model that balances variance explained by your predictors and overfitting. While you may use this for a small number of models, it is inadvisable to fit many models and select the model with the highest $R^2_{adj}$ value. (I would personally recommend building two or three models with which you are comfortable and comparing the $R^2_{adj}$ values, but no more.)

I may also recommend using Mallow's $C_p$, which is a statistic easily generated in many computer softwares. Mallow's $C_p$ can be calculated for multiple models (much like $R^2$ or $R^2_{adj}$) and you would want to select the model where $C_p$ comes closest to the number of predictors $p$ in that particular model. (For example, if model 1 had four predictors and model 2 had seven, you would select model 1 if $C_{p,1}$ was closer to 4 than $C_{p,2}$ was to 7, where $C_{p,i}$ is the value of Mallow's $C_p$ for model $i$.)

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks @Matt - I think using the adjusted r-squared is going to be the simplest and most easily defensible strategy here, though I'm also familiar with Mallow's. The adjusted r-squared for the model fit to the monthly and to the original (unaggregated) data is 0.82, and 0.802, which seems very consistent with what I am hoping to show. $\endgroup$ – Gabe Jul 6 '16 at 18:34
  • $\begingroup$ if you calculate adjusted r square for the example $1 - (1-0.910)\times\frac{41-1}{41-19-1}=0.828$. This is very close to the R square from the unit level model. probably a coincidence/quirk but I found it interesting. $\endgroup$ – probabilityislogic Jun 15 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.