4
$\begingroup$

The formula for the true Y which we try to estimate is $$Y = f(X) + \epsilon$$ I wonder why the $\epsilon$ is in there?

My thoughts: The true function would be the one that has the full predictor matrix available, including all relevant predictors for Y. Then when trying to estimate the true Y we use our available predictors in an incomplete predictor matrix, as we don't have all necessary predictors for the true $f(X)$. That would mean that our estimated function is never the true function, because, which would mean we would always have an error term, because of those unobserved predictors.

Therefore I'd imagine the formulas to be: $$Y = f(X)\\ \hat{Y}= f(X)+\epsilon$$, which would mean that the true Y is not a function of an error term. How should it if the error term is created by unobserved predictors, which is not the case for the true f(X) as that theoretical predictor matrix is imagined to be complete (no unobserved predictors).

So the question is: What am I getting wrong? Why should the true Y be a function of the error term?

$\endgroup$
2
  • $\begingroup$ Your question is close to religion no? Because nothing prove you can predict perfectly knowing everything (look en.wikipedia.org/wiki/Hidden-variable_theory). $\endgroup$
    – PauZen
    Jul 24, 2019 at 14:55
  • $\begingroup$ @PauZen Ah, I see. I thought the true $Y$ was a function of a perfectly everything-knowing predictor matrix.. $\endgroup$
    – thebilly
    Jul 24, 2019 at 15:33

2 Answers 2

4
$\begingroup$

The answer to this question relies on philosophy and it relates to whether $Y$ is deterministic, or whether there is an element of randomness.

It's fine if you want to believe for a particular problem that there is some set of covariates that would allow you to perfectly predict, so $Y=f(X)$.

However, since you will generally never observe all predictors, it's a moot point and you might as well start modeling with $Y=f(X)+ \epsilon$ as an approximation of the process that generates $Y$, up to the limits of what is knowable. For all intents and purposes, I am generally fine with pretending like the true $Y$ is not deterministic, that there are limits to knowledge and prediction of the future and that in some circumstances there may always be a random component that makes it impossible to perfectly predict $Y$.

$\endgroup$
2
  • $\begingroup$ I see. Thank you. So how is the theoretical function of the true $Y$ defined? (Not mathematically speaking but conceptually speaking.) $\endgroup$
    – thebilly
    Jul 24, 2019 at 15:31
  • $\begingroup$ This depends entirely on context and if the process that generates $Y$ is deterministic or has an element of randomness. What if Y is flipping coin? It's probably theoretically possible that perfect predictions can be made with enough knowledge about the forces applied when it leaves your hand. What if Y is whether or not I will die in the next year? Do I have free will to make choices? If so, then perfect prediction of what will happen in the next year is impossible, unless someone has the intent of killing me next year, but then only that person would be able to perfectly predict my death. $\endgroup$
    – jsk
    Jul 24, 2019 at 15:57
1
$\begingroup$

Your regression is modeling a conditional expected value of some response distribution. Therefore, your observed value is the expected value of the response distribution, plus some variation from the error term.

$$E[y\vert X] = \hat{y} = f(X)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.