Return to Answer

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. Overfitting risk is especially great if the model creators cannot explain how they got their formula except saying "it works well empirically", or some other weak, after-the-fact rationalization of a good result.

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.
• The model can predict data which is significantly dissimilar from what it was fit to.

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. Overfitting risk is especially great if the model creators cannot explain how they got their formula except saying "it works well empirically", or some other weak, after-the-fact rationalization of a good result.

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model can predict data which is significantly dissimilar from what it was fit to.
• The model can function under practical conditions - for example, predicting the future based only on past data, or predicting new customers' behavior based on old ones. Certain types of train-test splitting can help validate this.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. You should be especially suspicious if the authors can't explain how they got their formula except saying "it works well empirically".

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.
• The model can predict data which is significantly dissimilar from what it was fit to.

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. Overfitting risk is especially great if the model creators cannot explain how they got their formula except saying "it works well empirically", or some other weak, after-the-fact rationalization of a good result.

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.
• The model can predict data which is significantly dissimilar from what it was fit to.

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data and build a formula that fits well to training data, but fails to generalize to new data. You should be especially suspicious if the authors can't explain how they got their formula except saying "it works well empirically".

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.
• The model can predict data which is significantly dissimilar from what it was fit to.

No! The moment you look at all your data to see patterns and determine your model structure, you begin to risk overfitting. Whatever method was used to determine the transformations and variables included in the model, that method is itself part of the modeling process, a part that is just as susceptible to overfitting as the numerical algorithm used to fit the model. That's true even (and especially) if the method used was a person trying different regression formulas until one fit the data to their satisfaction.

Overfitting is a very general phenomenon. It's seeing patterns in your data that don't generalize. As such, it's not something that only a computer can commit. A human being can also see spurious patterns in data, think of a formula that describes these patterns, then get a nice fit to training data that will not generalize to new data. You should be especially suspicious if the authors can't explain how they got their formula except saying "it works well empirically".

That said, the risk we are discussing here is a risk and not a certainty. Plenty of people look at all of their data and still manage to build great models that generalize well. I prefer not to be too formalistic and dismissive when critiquing an analysis, but try to identify what factors make the model in question more or less likely to generalize. Factors that increase the likelihood of a model generalizing include:

• The model can predict data it hasn't been fit to - even if the model structure was built based on all the data.
• The model formulas have a structure that is consistent with or reinforced by previous well-established results and theories.
• Simplicity in the model structure.
• The model can predict data which is significantly dissimilar from what it was fit to.