# A Fundamental Question about Statistical Learning

In statistical learning (many textbooks), we assume that the data $Y$ is generated by $Y=f(X)+\epsilon$, where $X$ are predictors and $\epsilon$ is some random noise. Then the problem becomes: using various methods to find an estimates of $f$, i.e., $\hat{f}$ such that the expected mean square error on the test set is minimized. We know that we can decompose the mean square error into reducible error, which is related to $f-\hat{f}$ and irreducible error, which is related to $Var[\epsilon]$. It is clear that if you can have $\hat{f}=f$, then the reducible error become 0.

My question is: why there should be a "true" function $f$ such that we want to found an estimate $\hat{f}$ for and we wish it is close to $f$?

Consider the following man-made example. I generate my data set using the deterministic function $Y=g(X_1,X_2)$. So no random term here. Then I give the data set $(Y,X_1)$ to a friend, i.e., I omit the variable $X_2$. Hence, the data set $(Y,X_1)$ looks random to my friend, e.g., he might see that for the same value of $X_1$, it actually corresponds to 2 different value of $Y$. Then I ask him to find the best way to predict $Y$. In this example, to my friend, if he adopts the idea as the statistical learning textbook, i.e., he should believe that the data is generated from $Y=f(X_1)+\epsilon$. My question is, what is the "true" $f$ in this case, and is that meaningful to talk about "true" $f$?

I believe this is an important conceptual question, since in reality, I could always assume that every data set I saw is made by somebody who use a completely deterministic function and I just observe part of his set of predictors.

To me, the notion of "true" $f$ and "try to find an estimates $\hat{f}$ of true $f$" is redundant and meaningless. From a practical perspective, my goal is just to find some function $g$, such that $g(X)$ gives me the minimum expected mean square error on the test set.

• cf. "omitted variable bias"
– Sycorax
Jan 6, 2016 at 19:34
• So in practice, in every regression model, we must have omitted variable bias, since we never know what important stuff we have dumped to the $\epsilon$. So in practice, the notion of "consistency" is useless, since the "true data generating process" will absolutely not come from any parametric function we assumed Jan 6, 2016 at 20:32
• Maybe. In sufficiently controlled experiments, it's plausible to believe that there is no omitted variable bias.
– Sycorax
Jan 6, 2016 at 20:44
• It seems that you're thinking along the lines of the well known quote from George Box (or whoever): "All models are wrong, some are useful"... Jan 15, 2016 at 16:52

Since you only tell him, that Y is only generated by one variable he has, generally, no chance to find the true underlying generating function which is $g$.
• I think the point I want to make is that: when we do this regression type of thing, we don't have to say that our ultimate goal is to find a function such that it is close to the true function $f$. There are no such true $f$. What we do is just find A function such that it explains the data well in the sense that this function performs well on the test data set than some other functions. We never know how close the function we pick is close to the "true" function, since there might be no such true function Jan 6, 2016 at 20:29