I've been reading up on data snooping, and how it can mean the in-sample error does not provide a good approximation of the out-of-sample error.
Suppose we are given a data set $(x_1,y_1),(x_2,y_2),...,(x_n,y_n)$, which we plot, and observe what appears to be a quadratic relationship between the variables. So we make the assumption that $$ y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon, $$ where $\varepsilon$ is a variable representing noise.
Isn't this data snooping? We have let the data affect our model. So what implications does this have for the coefficients $\beta_0,\beta_1,\beta_2$ that we find; can they be considered reliable for making future predictions with different input variables?
I ask because there are countless notes/articles/books/etc.. on regression where they recommend looking at the data and then choosing a model that looks like it will fit well with the data. For example, here the author has some data, tries a linear model, and upon finding it unsatisfactory, he moves to a quadratic model which better fits the data. Similarly, here, people are discussing log transformations and the original poster is given the following advice:
If there is no theory to guide you, graphical exploration of the relationship between the variables, or looking at fitted vs observed plots both ways will tell you which model is appropriate.
So when we base our model on an observation of the plotted data, is this data snooping or not? If it isn't, then could someone give an explanation why this isn't data snooping?
If it is data snooping, then:
- What are the consequences of this on the out-of-sample performance?
- What should we do to avoid/overcome the data snooping issue in a regression model so that we will have good out-of-sample performance?