0
$\begingroup$

Due to limited data amount (100 observations) this model is built by full dataset with 2 independent variables. Once the functional form is determined, the data is split into training and test datasets, and refit the model on the training set and test the model on the test set. Can this approach avoid overfitting?

update: Let me put it in another way. The modelers built the model based on 10 years data. Once the model is done. The modelers split the data into training (first 8 years) and test (last 2 years) datasets. They refit the model on the training set, and test on the test set. Once they saw no big differences, they concluded that there is no overfitting. Is there any logical flaw?

Thanks for any suggestion.

update2:

Sorry for any confusion. Let me put it in a more specific way.

The target variable is a company size with 10 years monthly data. They have ten independent variables, like DOW, GDP growth, PMI, and so on. Due to limited data amount, the goal is to build a linear regression with 2 or 3 variables. They used the full dataset to get the final model with 2 variables. These 2 variables were selected based on business sense. Basically, they tried different combinations to get the best performance on the full data set. And they were just 2 linear terms, no transformation. Then they refit and model on the training set (first 8 years data) by using the selected 2 variables, and used the refitted model to test on the testing dataset (last 2 years data). They concluded there is no overfit. Without considering the metrics they used, does this process make sense to get the conclusion?

$\endgroup$
  • 1
    $\begingroup$ It sounds double dipping - what do you mean by "the functional form"? Why not using the training set to determine the functional form and fit the model then cross validate on the test set? $\endgroup$ – Junpeng Lao Oct 25 '15 at 16:15
  • $\begingroup$ When I say the functional form, I mean use the transformation and variables determined by full data set. I think the more appropriate way is to use the training to get the model, but the modelers argued that they tested the results by this way and no overfitting was detected. $\endgroup$ – Eric Oct 25 '15 at 16:21
  • 1
    $\begingroup$ While I agree that "training and test" splits do not, in and of themselves, avoid overfitting, they are an essential step in the process of understanding whether or not the model is overfitted. This discussion of overfitting in Wikipedia is pretty good... en.wikipedia.org/wiki/Overfitting All of the metrics and heuristics for overfitting notwithstanding (e.g., AICs, BICs, regularization, etc.) with so few observations, you can never get to any level of confidence that you have not overfit the model. $\endgroup$ – Mike Hunter Oct 25 '15 at 16:58
  • $\begingroup$ @DJohnson I agree, yet I don't think we should be too pessimistic without knowing more. Most of the scientific revolution was based on fitting models to experimental datasets consisting of around 10-100 observations. Data size and statistical validation are factors but do not give a complete picture of overfitting risk. $\endgroup$ – Paul Oct 25 '15 at 17:03
  • 1
    $\begingroup$ @max That is a true and fair point. Tversky and Kahneman's Prospect Theory is another, good example of that. In all of those cases, theory trumped empirical and/or statistical observation. This doesn't change the fact that there is still a low level of statistical power or confidence in the findings. The bottom line with overfitting is apophenia, finding spurious patterns in data -- small, large or massive. Unfortunately for the history of science there are at least as many examples of this as there are of cases where theory was correct. $\endgroup$ – Mike Hunter Oct 25 '15 at 17:30
3
$\begingroup$

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. Overfitting risk is especially great if the model creators cannot explain how they got their formula except saying "it works well empirically", or some other weak, after-the-fact rationalization of a good result.

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

  • The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
  • The model can predict data which is significantly dissimilar from what it was fit to.
  • The model can function under practical conditions - for example, predicting the future based only on past data, or predicting new customers' behavior based on old ones. Certain types of train-test splitting can help validate this.
  • The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
  • Simplicity in the model structure.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.