3
$\begingroup$

I am trying to read the book "Computer Age Statistical inference", which is available here: https://web.stanford.edu/~hastie/CASI_files/PDF/casi.pdf

At the very start of the book (from page 4) there is a linear regression example with "standard error" estimates of the predictions, and it has completely confused me.

The setup is the following. They have a linear regression model,

$$𝑦 = \beta_0 + \beta_1x$$

and using "least squares" they deduce that the parameters have values

$$\beta_0 = 2.86, \beta_1 = -0.079 $$

Ok. Then they have a table of possible $x$ values, with the corresponding predicted $y$ and the "standard error" for each prediction, which is different for each one (this is what I don't understand).

To give two examples, they have:

$$x=20, y=1.29, stderror=0.21$$

$$x=30, y=0.5, stderror=0.15$$

The standard errors are different for the different examples.

Now, a few pages earlier they discuss computing the mean of a sample, and calculating the standard error according to the formula:

$$se = [\sum_{i=1}^n \frac{(x_i-\bar x)^2}{n(n-1)}]^{0.5}$$

They state that in the linear regression case, they compute the standard errors using an "extended version" of the above formula - but never actually say what it is. I don't understand how they have calculated the "standard error" for the individual examples in the linear regression case, and why it is different for each example. What is the formula?

Reading the wikipedia page on the standard error makes me think the se is just the standard deviation of the sample - but this doesn't fit in with what they have done here.

Any help appreciated.

$\endgroup$

2 Answers 2

4
$\begingroup$

You can download their example from https://web.stanford.edu/~hastie/CASI_files/DATA/kidney.txt and easily replicate their results.

> kidney <- read.table("kidney.txt", header=TRUE)
> str(kidney)
'data.frame':   157 obs. of  2 variables:
 $ age: int  18 19 19 20 21 21 21 22 22 22 ...
 $ tot: num  2.44 3.86 -1.22 2.3 0.98 -0.5 2.74 -0.12 -1.21 0.99 ...
> fit <- lm(tot ~ age, data=kidney)
> fit$coefficients
(Intercept)         age 
 2.86002680 -0.07858842 

As to standard errors, standard errors for fitted values, $\text{se}(\hat{y})$, are differente from coefficients' standard errors, $\text{se}(\hat\beta)$.

The model matrix $X$ is:

> X <- model.matrix(fit)
> head(X)
  (Intercept) age
1           1  18
2           1  19
3           1  19
4           1  20
5           1  21
6           1  21

Putting $S=(X^TX)^{-1}$, $\text{cov}(\hat\beta)=\sigma^2_yS$ (see my answer to this question). Given a single fitted value, $\hat{y}_h$ and the corresponding $h$th row of $X$, e.g. $$y_1=2.44,\qquad x_1=\begin{bmatrix}1 \\ 18\end{bmatrix}$$ the variance of $\hat{y}_h$ is: $$\text{var}(\hat{y}_h)=\text{var}(x_h^T\hat\beta)=x_h^T\text{cov}(\hat\beta)x_h=x_h^T(S\sigma^2_y)x_h =\sigma^2_y(x_h^TSx_h)$$ You estimate $\sigma^2_y$ by the residual mean square, RMS, the stardard error of $\hat{y}_h$ is: $$\text{se}(\hat{y}_h)=\sqrt{RMS(x_h^TSx_h)}$$ and it depends on $x_h$.

When there is only one independent variable, $$S=(X^TX)^{-1}=\frac{1}{n\sum(x_i-\bar{x})^2} \begin{bmatrix}\sum x_i^2 & -\sum x_i \\ -\sum x_i & n \end{bmatrix}$$ and \begin{align*} x_h^T(X^TX)^{-1}x_h &=\frac{\sum x_i^2-2x_hn\bar{x}+nx_h^2}{n\sum(x_i-\bar{x})^2}=\frac{\sum x_i^2 -n\bar{x}^2+n(x_h-\bar{x})^2}{n\sum(x_i-\bar{x})^2}\\ &=\frac1n+\frac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2} \end{align*} (Remember that $\sum(x_i-\bar{x})^2=\sum x_i^2-n\bar{x}^2$).

The "extended version of formula (1.2)" (which is just the standard error of a mean) is: $$\text{se}(\hat{y}_h)=\left[RMS\left(\frac1n+\frac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)\right]^{\frac12}$$ BTW, this is how confidence bands are calculated.

See Kutner, Nachtsheim, Neter & Li, Applied Linear Statistical Models, McGraw-Hill, 2005, §2.4, or Seber & Lee, Linear Regression Analysis, John Wiley & Sons, 2003, §6.1.3.

In R:

> S <- solve(t(X) %*% X)
> RMS <- summary(fit)$sigma^2
> x_h <- matrix(c(1, 20), ncol=1)             # first standard error in Table 1.1
> y_h_se <- sqrt(RMS * (t(x_h) %*% S %*% x_h)); y_h_se
          [,1]
[1,] 0.2066481
> x_h <- matrix(c(1, 80), ncol=1)             # last standard error in Table 1.1
> y_h_se <- sqrt(RMS * (t(x_h) %*% S %*% x_h)); y_h_se
         [,1]
[1,] 0.420226

EDIT

If you are interested in the standard error of $\hat{y}_{h(new)}=\hat\alpha+\hat\beta x_{h(new)}$, when $x_{h(new)}$ is a new observation, you do not know what $\hat{y}_h$ would be in a regression on $n+1$ points. Different samples would yield different predictions, so you have to take into account the deviation of $\hat{y}_{h(new)}$ around $\hat{y}_h=\hat\alpha+\hat\beta x_h$: $$\text{var}[y_{h(new)}-\hat{y}_h]=\text{var}(y_{h(new)})+\text{var}(\hat{y}_h)$$ So the variance of your prediction has two components: the variance of $y$, which you estimate by RMS, and the variance of the sampling distribution of $\hat{y}_h$, $RMS(x_h^TSx_h)$:

$$RMS + RMS\left(\frac1n+\frac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)$$ The "extended version of formula (1.2)" turns into: $$\text{se}(\hat{y}_{h(new)})=\left[RMS\left(1+\frac1n+\frac{(x_{h(new)}-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)\right]^{\frac12}$$ See Kutner, Nachtsheim, Neter & Li, Applied Linear Statistical Models, McGraw-Hill, 2005, §2.5, or https://online.stat.psu.edu/stat501/lesson/3/3.3.

$\endgroup$
5
  • $\begingroup$ This answer really cleared things up for me. One point I wasn't understanding was why $\text{cov}(\hat\beta)=\sigma^2_yS$ . For other people confused, this document stat.purdue.edu/~boli/stat512/lectures/topic3.pdf really goes through the details of how to calculate the covariance of the estimator. $\endgroup$
    – John Smith
    Commented Sep 9, 2020 at 6:09
  • $\begingroup$ One point of clarification: so in your answer $x_h$ is a row of our training data matrix X. But in principle we could have $x_h$ be a future data point. So then in this case, $+- 2 se(\hat{y}_h)$ will trap our prediction with 95% confidence, assuming the model is correct. Is this right? (Apologies for multiple edits - still learning how to do formatting) $\endgroup$
    – John Smith
    Commented Sep 9, 2020 at 6:10
  • $\begingroup$ In fact, the exact language in the book is "...two standard errors, giving them about 95% chance of containing the true expected value", and the picture in the book shows this is for future values as well - all contingent on model correctness of course. $\endgroup$
    – John Smith
    Commented Sep 9, 2020 at 6:32
  • $\begingroup$ @JohnSmith As to $\text{cov}(\hat\beta)$ you could also loot at stats.stackexchange.com/questions/486464/… $\endgroup$
    – Sergio
    Commented Sep 9, 2020 at 6:40
  • $\begingroup$ @JohnSmith As to the standard error of new predictions, I've edited my answer. $\endgroup$
    – Sergio
    Commented Sep 9, 2020 at 8:01
2
$\begingroup$

The predicted value at $X=x$ is $\hat\mu=\hat\beta_0+\hat\beta_1x$. This is a function of a known constant, $x$, and random variables $(\hat\beta_0, \hat\beta_1)$. The standard error of $\hat\mu$ is its standard deviation, which is a function of the standard deviation of $(\hat\beta_0, \hat\beta_1)$

Specifically, the variance of $x\hat\beta$ is $$x^2\mathrm{var}[\hat\beta_1]+2x\mathrm{cov}[\hat\beta_1,\hat\beta_0]+ \mathrm{var}[\hat\beta_0].$$

This depends on $x$, so it's different for each observation. Since we know $x$ and have a good estimator of the variance-covariance matrix of $\hat\beta$ we can estimate it.

The reason for the simplified formula you quote is that the linear algebra becomes simpler if the mean of $X$ is zero, so that $\hat\beta_0$ and $\hat\beta_1$ are uncorrelated. You can arrange that by transforming $x$ to $x-\bar x$.

$\endgroup$
7
  • $\begingroup$ I'm sorry, I don't understand. Why are the beta coefficient values random variables? This is never stated in the example. What is their distribution? Where does it come from? In the example they are solved constants, beta = (2.86,-0.079). $\endgroup$
    – John Smith
    Commented Sep 8, 2020 at 16:42
  • $\begingroup$ Oh, because the beta values are functions of the data, and the data is random, they are random too? $\endgroup$
    – John Smith
    Commented Sep 8, 2020 at 16:58
  • $\begingroup$ @JohnSmith The $\beta$ coefficient are not random variables (unless you are doing bayesian inference). Their estimators $\hat\beta$ are random variables, because $y$ is random and $\hat\beta=(X^TX)^{-1}X^Ty$. $\endgroup$
    – Sergio
    Commented Sep 8, 2020 at 21:50
  • $\begingroup$ @JohnSmith Yes -- though 'beta coefficient' is an unfortunate name for $\hat\beta$ even if it is very common, precisely because of making this confusion likely. $\endgroup$ Commented Sep 9, 2020 at 2:33
  • $\begingroup$ This was a great answer. If I'm looking at the two predictions on two points in the data (like the setup above, with two different standard errors of prediction), then should I assume that the distribution are independent? $\endgroup$
    – ABC
    Commented Nov 22, 2021 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.