4
$\begingroup$

I am currently reading some notes on linear and polynomial regression. The notes say the following:

The linear model is $$\hat{f}_L (X) = \beta_0 + \beta_1 X$$ The quadratic model is $$\hat{f}_Q(X) = \beta_0 + \beta_1 X + \beta_2 X^2$$

Notice that the $\hat{f}$ don't have an error term, $\epsilon$, as would be expected for regression models. Why don't these models have an error term?

$\endgroup$

3 Answers 3

9
$\begingroup$

Have a look at wikipedia

The hat on top of a variable (e.g. $\hat{f}_L(X)$) typically indicates that the variable is an estimator of some observable. Namely, it is a function of a sample from a random variable, which can be used to estimate something of real world relevance, such as Celsius temperature given Farenheit temperature measurement. This view is derived from a more general view concerning relationships of random variables themselves. There, as you see in wiki first equation, the errors are present. I would say that it is more precise to call what you have above a linear estimator and not a linear model, but I am not the author of the book you are reading

$\endgroup$
1
  • 1
    $\begingroup$ +1 So far, this is the sole answer that actually answers the question. $\endgroup$
    – whuber
    Commented Aug 20, 2021 at 14:03
8
$\begingroup$

For a linear model there are two equivalent ways to write the model:

$$Y = X\beta + \epsilon$$

and

$$E(Y \mid X) = X\beta$$

Where $\epsilon$ is the error term and $E()$ denotes expected value (long-term average). $X\beta$ is the linear predictor, i.e., weighted sum of predictors. When writing the model in terms of expected values, the error term need not be included.

For Gaussian errors (the usual linear model) I prefer an all-encompassing statement of the model. Here $\Phi$ is the Gaussian cumulative distribution function and $\sigma$ is the residual standard deviation.

$$\Pr(Y \leq y \mid X) = \Phi \left( \frac{y - X\beta}{\sigma} \right)$$

If the model is not a linear model, there need not be an error term anywhere. For example when $Y$ is binary the model is almost always stated as a probability statement similar to the above probability statement.

$\endgroup$
1
$\begingroup$

There are two ways to write the equation of a regression model. One describes the actual relationship between X and Y, and the other describes the estimated relationship.

This equation

$$ Y_i= \beta_0 + \beta_1 X_i +\epsilon_i$$

Means "For each observation "i" in the dataset, that observation's Y value equals (exactly!) $\beta_1$ multiplied by that observation's X value, plus $\beta_0$, plus some value $\epsilon_i$." Note that this equation will always be true no matter how good or bad your model is.

By contrast, the equation

$$ \hat{Y_i}= \beta_0 + \beta_1 X $$

means "For each observation "i" in the dataset, we estimate (that's what the hat on Y means) that this observation's Y value equals $\beta_1$ multiplied by that observation's X value, plus $\beta_0$."

Obviously, we know that this estimate probably going to be wrong for most observations (that's why it's an estimate). And in any given dataset it is going to be wrong by $\epsilon_i$ for each observation. But if all we want to talk about is our estimate, then this is a fine way to write the model.

$\endgroup$
1
  • 2
    $\begingroup$ Shouldn't the $\beta$s in the second equation also have hats because they're estimates? $\endgroup$ Commented Aug 20, 2021 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.