# How to use residuals to choose the best model parameters and avoid over-fitting?

A common way to avoid an over-fitting is to train a model on one set and then to check the error on the validation set. If the out-of-sample error is much larger than the in-sample error than we have an over-fitting and therefore we need to suppress the expressive power of the model (either by using a model with a smaller number of parameters or by implying penalties proportional to values of the parameters on the model).

In fact the procedure is based on comparison of two errors (in-sample and out-of-sample). Is it possible to generalize this procedure? For example, for any value of the model parameters we can calculate the residuals (differences between the target and the predicted values). Then we somehow compare the residuals for two alternative models and in this way (some how) we decide what set of model parameters is better.

We want to do it in such a way that avoid over-fitting. Obviously the simplest (naive) way to do it is to calculate some average of residuals and then choose the model providing smaller residuals on average, but this obviously do not prevent us from an over-fitting.

It looks to me that it should go in the direction of variation of residuals. We want to know not only what are their values on average but also what is there variance. For example if average is small but variance is large we can assume that such a small average is observed just by chance.

Maybe it should go in the direction of statistical significance. We can say that model B is better than model A not only if the mean of the residuals of the model B is smaller, this decrease should also be statistically significant.

• Are you referring to mean squared error (MSE)? The average of the residuals is zero. I think the generalization of comparing in- and out-of-sample would be cross-validation. – Frans Rodenburg Oct 28 '17 at 14:07
• I assume you mean something other than AIC (or BIC) or cross-validation, which kind of do what you ask for? It might help, if you explained why you are looking for something other than these existing approaches. – Björn Jan 10 '18 at 12:55
• The reason why I am looking for something beyond standard cross-validation is that it seems to me to be a bit "artificial" and I guess that it is not the best use of data. We split the data set into two parts and then compare the corresponding two errors. But why not to split it into 5 sets and make decision based on these 5 error. Or we can go even further, why not to have just one observation per data set (so we do the maximal possible number of splits) and then we make decision (about what model is better) based on all the errors (or residuals, if we have one point per dataset). – Roman Jan 10 '18 at 15:21
• Cross-validation allows you to perform any of the splits you mention; is this what you see as (part of the) problem with cross-validation or is it a desirable feature of an answer to your question? – jbowman Jan 10 '18 at 18:11
• It is a desirable feature. I want to have a solution that is flexible in terms of how we split the data. It looks to me that the best split (in terms of utilizing the data) is the "maximal" one (one split is one observation). What I want is a rigorous way to aggregate the errors (or residuals, in case of the "maximal" split) into one measure that is used to select a model such that we avoid overfitting. – Roman Jan 11 '18 at 7:51

I am not sure that there are any systematic way to achieve residual benchmarking across all model types. Any model domain calls for different testing and what have you.

I just want to clarify a few concepts and share some others that are useful within the domain of modeling macroeconomics time series (econometrics modeling).

First, clarification... a model that has larger out-of-sample error vs. in-sample error is not an overfit model. This is by definition. Unless by pure fortuitous randomness, a model typically will make smaller errors on the learning sample where it fully captures the data info vs. the out-of-sample where it does not capture any info. The only typical situation is when the out-of-sample period is a lot more stable than the in-sample one. Let's say your in-sample period is a monthly time series between 2005 and 2010, around the Financial Crisis. And, your out-of-sample period is from 2013 to 2016; it is not unlikely your residuals will be lower during the out-of-sample period. But, you have to understand why. It is not necessarily because your model is great. It is in good part, because the in-sample period was so much more volatile.

And, that is the challenge with residual analysis. Although, it is the best and most extensive tool there is to test a model... you have to pay close attention whether the relative out-of-sample model performance is a function of the relative volatility difference in the data or due to the model structural strength or weakness. And, only studying the data and the model with a real critical eye can discern the answer.

When comparing the in-sample vs. out-of-sample residual performance, there are a few metrics you can look at. One is the standard error of the model over those respective period. The latter does not work well when the size of your samples are very different (typically a lot smaller within the out-of-sample sample). Instead, you could just use the standard deviation of the residuals which is the same thing but not adjusted or in this case distorted by the different degrees of freedom within the in-sample larger sample vs. the smaller out-of-sample sample. There are other measures catered for this type of exercise including the Mean Absolute Error (MAE) and the Mean Absolute Percentage Error (MAPE). Depending on what your dependent variable is either the MAE or MAPE may be more relevant. And, you compare those between the in-sample and out-of-sample results.

This type of analytical metrics (residual standard deviation, MAE, MAPE) is typically more useful than any information criterion (AIC, BIC, etc.) that are at times difficult to interpret.

Also, the mentioned analysis (comparing MAE, etc.) is a lot more useful when not only comparing one model's in-sample vs. out-of-sample performance, but, when comparing two or more competing models. And, the MAE on the out-of-sample can provide you a lot of info in terms of selecting your best model.

In addition to measuring and comparing MAEs, I would also compare MEs or MPEs. That's because when you look at absolute errors you can mask the upward or downward bias of a model during the out-of-sample period. We know a model should have a mean error of zero over the in-sample period. But, we also know it won't be the case during the out-of-sample period. And, it is important to measure that bias in either direction.

So, your best model is typically the one that will have the lowest MAE, MAPE and also lower ME or MPE (smaller bias). Notice that this best model is not derived by simply looking at the model with the highest R Square and lowest RMSE within the in-sample period. Many get obfuscated by models with very high R Square on the learning sample without even testing the model appropriately on a series of out-of-sample tests.

Bailey, Borwein, and Lopez de Prado tackled this issue not by examining residuals, but by computing a probability that the backtest has been overfit. They use the amount of data available and the number of model configurations together with a model evaluation metric like Sharpe ratio. See The Probability of Backtest Overfitting and related papers listed at SSRN. These authors also employ symmetric cross-validation as some commenters have mentioned.

I think there is a confusion for OP. OP is mixing two concepts together when think about the performance of the model. These two concepts are:

• The loss for the entire data set
• The residual for each of the data points.

These two things are very different. The loss for the entire data set usually is just a scalar / number. For example, it can be $1234.0$, which is sum of the squared errors. We want to minimize this number to have a good model. And this number is usually used to select the best model or decide if we are doing overfitting. But keep in mind that this is a single number.

On the other hand, the residual are many numbers. Number of the residuals is equal to number of data points. When do model selection, we usually do not think about the many numbers but think about one number. For example, if we feel we want to minimize residual variance. Then, we can use residual variance for the entire data set as the loss to minimize.

In sum, I suggest OP to think the difference between this two things: loss on the entire data set usually is one number and used for model selection. And the residual are many numbers, if we want to use it for model selection, we need to think about some "aggregation function" to make it into a single number / to calculate the loss in entire data set.

• you misunderstood the question. I perfectly know what is the difference between the residuals and the loss function. – Roman Jan 12 '18 at 7:15