In statistical learning (many textbooks), we assume that the data $Y$ is generated by $Y=f(X)+\epsilon$, where $X$ are predictors and $\epsilon$ is some random noise. Then the problem becomes: using various methods to find an estimates of $f$, i.e., $\hat{f}$ such that the expected mean square error on the test set is minimized. We know that we can decompose the mean square error into reducible error, which is related to $f-\hat{f}$ and irreducible error, which is related to $Var[\epsilon]$. It is clear that if you can have $\hat{f}=f$, then the reducible error become 0.

My question is: why there should be a "true" function $f$ such that we want to found an estimate $\hat{f}$ for and we wish it is close to $f$?

Consider the following man-made example. I generate my data set using the deterministic function $Y=g(X_1,X_2)$. So no random term here. Then I give the data set $(Y,X_1)$ to a friend, i.e., I omit the variable $X_2$. Hence, the data set $(Y,X_1)$ looks random to my friend, e.g., he might see that for the same value of $X_1$, it actually corresponds to 2 different value of $Y$. Then I ask him to find the best way to predict $Y$. In this example, to my friend, if he adopts the idea as the statistical learning textbook, i.e., he should believe that the data is generated from $Y=f(X_1)+\epsilon$. My question is, what is the "true" $f$ in this case, and is that meaningful to talk about "true" $f$?

I believe this is an important conceptual question, since in reality, I could always assume that every data set I saw is made by somebody who use a completely deterministic function and I just observe part of his set of predictors.

To me, the notion of "true" $f$ and "try to find an estimates $\hat{f}$ of true $f$" is redundant and meaningless. From a practical perspective, my goal is just to find some function $g$, such that $g(X)$ gives me the minimum expected mean square error on the test set.

  • 1
    $\begingroup$ cf. "omitted variable bias" $\endgroup$
    – Sycorax
    Jan 6, 2016 at 19:34
  • $\begingroup$ So in practice, in every regression model, we must have omitted variable bias, since we never know what important stuff we have dumped to the $\epsilon$. So in practice, the notion of "consistency" is useless, since the "true data generating process" will absolutely not come from any parametric function we assumed $\endgroup$
    – KevinKim
    Jan 6, 2016 at 20:32
  • 1
    $\begingroup$ Maybe. In sufficiently controlled experiments, it's plausible to believe that there is no omitted variable bias. $\endgroup$
    – Sycorax
    Jan 6, 2016 at 20:44
  • $\begingroup$ It seems that you're thinking along the lines of the well known quote from George Box (or whoever): "All models are wrong, some are useful"... $\endgroup$ Jan 15, 2016 at 16:52

1 Answer 1


There are several issues here: the space your friend shall minimize in is uncountably. Hence, if he is not really lucky and guesses the correct function, there is no chance in minimizing the expression. Thus, one chooses a certain class in that one seeks to minimize. However, typically the class of function is biased. Like the Class of linear functions. In such a class of functions, one searches

Since you only tell him, that Y is only generated by one variable he has, generally, no chance to find the true underlying generating function which is $g$.

However, with the wrong assumption of a process depending only on one variable (i.e. assuming that an underlying f exists), he can try to find a function that explains the data reasonably well

  • $\begingroup$ I think the point I want to make is that: when we do this regression type of thing, we don't have to say that our ultimate goal is to find a function such that it is close to the true function $f$. There are no such true $f$. What we do is just find A function such that it explains the data well in the sense that this function performs well on the test data set than some other functions. We never know how close the function we pick is close to the "true" function, since there might be no such true function $\endgroup$
    – KevinKim
    Jan 6, 2016 at 20:29
  • $\begingroup$ It depends, in your artificial problem there is a true g however in general there is none, thus one tries to find an underlying model that describes the relevant effects. Let me quote George Box: “all models are wrong, but some are useful.” That's the Spirit of statistical learning $\endgroup$ Jan 6, 2016 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.