Does this approach avoid overfitting?

Question

Due to limited data amount (100 observations) this model is built by full dataset with 2 independent variables. Once the functional form is determined, the data is split into training and test datasets, and refit the model on the training set and test the model on the test set. Can this approach avoid overfitting?

update: Let me put it in another way. The modelers built the model based on 10 years data. Once the model is done. The modelers split the data into training (first 8 years) and test (last 2 years) datasets. They refit the model on the training set, and test on the test set. Once they saw no big differences, they concluded that there is no overfitting. Is there any logical flaw?

Thanks for any suggestion.

update2:

Sorry for any confusion. Let me put it in a more specific way.

The target variable is a company size with 10 years monthly data. They have ten independent variables, like DOW, GDP growth, PMI, and so on. Due to limited data amount, the goal is to build a linear regression with 2 or 3 variables. They used the full dataset to get the final model with 2 variables. These 2 variables were selected based on business sense. Basically, they tried different combinations to get the best performance on the full data set. And they were just 2 linear terms, no transformation. Then they refit and model on the training set (first 8 years data) by using the selected 2 variables, and used the refitted model to test on the testing dataset (last 2 years data). They concluded there is no overfit. Without considering the metrics they used, does this process make sense to get the conclusion?

Answer 1

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. Overfitting risk is especially great if the model creators cannot explain how they got their formula except saying "it works well empirically", or some other weak, after-the-fact rationalization of a good result.

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model can predict data which is significantly dissimilar from what it was fit to.
• The model can function under practical conditions - for example, predicting the future based only on past data, or predicting new customers' behavior based on old ones. Certain types of train-test splitting can help validate this.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.