Using a simple ordinary least squares regression:

$ Y = \alpha + \beta \times X$

we can estimate the dependent variable $Y$ through the regression parameters of $\alpha \text{ and } \beta$.

In what way is the estimated $Y$ "better" than the original $Y$?

  • $\begingroup$ Better for whom or what? Measuring the coefficient of the law of gravity in a lab? Reporting the profit of your company to the tax office? Correcting for instrument error? $\endgroup$ – Paul Jan 12 '14 at 4:57

You wouldn't normally call the observed value an 'estimated value'.

However, in spite of that, the observed value is nevertheless technically an estimate of the mean at its particular $x$, and treating it as an estimate will actually tell us the sense in which OLS is better at estimating the mean there.

Generally speaking regression is used in the situation where if you were to take another sample with the same $x$'s, you would not get the same values for the $y$'s. In ordinary regression, we treat the $x_i$ as fixed/known quantities and the responses, the $Y_i$ as random variables (with observed values denoted by $y_i$).

Using a more common notation, we write

$$Y_i = \alpha + \beta x_i + \varepsilon_i$$

The noise term, $\varepsilon_i$, is important because the observations don't lie right on the population line (if they did there'd be no need for regression; any two points would give you the population line); the model for $Y$ must account for the values it takes, and in this case, the distribution of the random error accounts for the deviations from the ('true') line.

The estimate of the mean at point $x_i$ for ordinary linear regression has variance

$$\Big(\frac{1}{n} + \frac{(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2}\Big)\,\sigma^2$$

while the estimate based on the observed value has variance $\sigma^2$.

It's possible to show that for $n$ at least 3, $\,\frac{1}{n} + \frac{(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2}$ is no more than 1 (but it may be - and in practice usually is - much smaller). [Further, when you estimate the fit at $x_i$ by $y_i$ you're also left with the issue of how to estimate $\sigma$.]

But rather than pursue the formal demonstration, ponder an example, which I hope might be more motivating.

Let $v_f = \frac{1}{n} + \frac{(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2}$, the factor by which the observation variance is multiplied to get the variance of the fit at $x_i$.

However, let's work on the scale of relative standard error rather than relative variance (that is, let's look at the square root of this quantity); confidence intervals for the mean at a particular $x_i$ will be a multiple of $\sqrt{v_f}$.

So to the example. Let's take the cars data in R; this is 50 observations collected in the 1920s on the speed of cars and the distances taken to stop:

enter image description here

So how do the values of $\sqrt{v_f}$ compare with 1? Like so:

enter image description here

The blue circles show the multiples of $\sigma$ for your estimate, while the black ones show it for the usual least squares estimate. As you see, using the information from all the data makes our uncertainty about where the population mean lies substantially smaller - at least in this case, and of course given that the linear model is correct.

As a result, if we plot (say) a 95% confidence interval for the mean for each value $x$ (including at places other than an observation), the limits of the interval at the various $x$'s are typically small compared to the variation in the data:

enter image description here

This is the benefit of 'borrowing' information from data values other than the present one.

Indeed, we can use the information from other values - via the linear relationship - to get good estimates the value at places where we don't even have data. Consider that there's no data in our example at x=5, 6 or 21. With the suggested estimator, we have no information there - but with the regression line we can not only estimate the mean at those points (and at 5.5 and 12.8 and so on), we can give an interval for it -- though, again, one that relies on the suitability of the assumptions of linearity (and constant variance of the $Y$s, and independence).

| cite | improve this answer | |
  • $\begingroup$ +1. (But I am unable to demonstrate $\frac{1}{n} + \frac{(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2}\lt 1$ unless I assume $n \ge 3$ and not all $x_i$ are the same; even then, I can only show the relationship is $\le$, not $\lt$. :-) Why do you write that one cannot estimate $\sigma$ from the data? I thought this was routinely done by taking the root of the mean squared residual. I must not understand your claim correctly. $\endgroup$ – whuber Jan 10 '14 at 22:50
  • $\begingroup$ @whuber I made some changes. If you have any further commentary, it would be very welcome. $\endgroup$ – Glen_b Jan 11 '14 at 0:03
  • $\begingroup$ Thanks! (You really didn't need to work that hard. :-) $\endgroup$ – whuber Jan 11 '14 at 0:05

First, the regression equation is:

\begin{equation} Y_i = \alpha + \beta X_i + \epsilon_i \end{equation}

There is an error term, $\epsilon$. As it turns out, this error term is critical to answering your question. What, exactly, is the error term in your application? One common interpretation of it is "the influence of everything, other than $X$, which affects $Y$." If that is your interpretation of your error term, then $Y_i$ is the best measure of what $Y_i$ really is.

On the other hand, in some rare cases we interpret the error term as being exclusively measurement error---the error induced by the operator error in using a scientific instrument or the error coming from the naturally limited precision of an instrument. In that case, the "real" value of $Y_i$ is $\alpha+\beta X_i$. In this case, you should use the OLS prediction of $Y_i$ instead of the actual value of $Y_i$ if $V(\epsilon_i)>V(\hat{\alpha}_{OLS}+\hat{\beta}_{OLS} X_i)$---that is if the variance of the error which comes from replacing $\alpha$ and $\beta$ with their OLS estimators is smaller than the variance of the measurement error.

| cite | improve this answer | |
  • $\begingroup$ Another example: If the true relationship between Y and X is non-linear, then alpha and beta, and thus Yest, are only a local linearization of a non-linear function. The error term will pick up effects that can not be captured by a linear fit. In such a case Yest can very well be a biased estimator of Y[i], i.e. the expected error if we made a new observation should not be zero. $\endgroup$ – Paul Jan 12 '14 at 4:55

The original value is not an estimate (except for the fact that it may have measurement error): It is the value of Y for a specific subject (e.g. person or whatever). The predicted value from the equation is an estimate: It is an estimate of the expected value of Y at a given value of X.

Let's make this concrete:

Let's say Y is weight and X is height. Let's say you measure and weigh a bunch of people. Let's say Jill is 5'0 and 105 pounds. That is her height and weight. The equation will give you a different predicted value of weight for a person who is 5'0". That is not the predicted value for Jill - you don't need to predict or estimate her weight, you know it to the precision of the scale. It is the predicted value of some "typical 5'0" person".

| cite | improve this answer | |
  • $\begingroup$ So can we assume that regression is some kind of clustering where the classes are found by the regression line? $\endgroup$ – Kare Jan 11 '14 at 0:01
  • $\begingroup$ No, no classes need be formed. It is a fitting of a least squares line. $\endgroup$ – Peter Flom Jan 11 '14 at 2:07

The equation should be $$\operatorname{E}(Y)=\alpha+\beta x$$; that is the expected value of $Y$ at the given value of $x$. So, if your model's right & you make enough observations of $Y$ at that value of $x$, it tells you what the average value of $Y$ will be. In the long run you'll do better making predictions using that average than the value you observed.

| cite | improve this answer | |
  • $\begingroup$ Thx for your answer! Could you please explain why I would "do better making predictions"? $\endgroup$ – Kare Jan 11 '14 at 0:03

Typically, OLS is typically not motivated by comparing the estimated response, $\hat{Y_i}$, to the observed response $Y_i$. Instead, if given a new set of values for the predictor value $X_{new}$, the OLS model predicts what the dependent variable would be $\hat{Y}_{new}$ in a typical case.

The point is that $\hat{Y}_i$ is typically not considered "better" than $Y_i$, but rather a more accurate reflection of what you expect $Y$ to be at a particular value for $X$.

However, there are situations when you may think $\hat{Y}_i$ more accurately reflects the truth than $Y_i$ (perhaps for an outlier arising from a malfunction in your data collection). This would be highly dependent on the details of your data.

| cite | improve this answer | |

Does this help? (It was what first came to my mind on reading the question.)

In statistics, the Gauss–Markov theorem, named after Carl Friedrich Gauss and Andrey Markov, states that in a linear regression model in which the errors have expectation zero and are uncorrelated and have equal variances, the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares (OLS) estimator. Here "best" means giving the lowest variance of the estimate, as compared to other unbiased, linear estimates. The errors don't need be normal, nor independent and identically distributed (only uncorrelated and homoscedastic). The hypothesis that the estimator be unbiased cannot be dropped, since otherwise estimators better than OLS exist.


| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.