"Centered" linear regression in point-slope form: pivot distribution and notation

Question

I'm a "pure math" probabilist who's been roped into teaching an undergraduate statistics course, despite little experience with statistics per se, and I'm trying to stay one chapter ahead of the students. So apologies in advance if I have mistakes in terminology, notation, etc., or miss things that should be obvious.

Anyway, the class is studying simple linear regression. The textbooks we're using (Pishro-Nik, Evans and Rosenthal) define it as assuming that the predictor $x$ and the response $Y$ are related by the following linear equation in slope-intercept form: $$Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \tag{1}$$ where $\epsilon_i \sim N(0, \sigma^2)$ are iid with $\sigma^2$ unknown. (Evans and Rosenthal call the parameters $\beta_1, \beta_2$ instead.) We use the usual estimator $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x}) (Y_i - \bar{Y})}{\sum (x_i - \bar{x})^2}$ for the slope, and then use it to estimate the $y$-intercept $\beta_0$ by $\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{x}$. Evans and Rosenthal go on to derive the distributions of pivotal quantities for $\hat{\beta}_1, \hat{\beta}_0$ which can be used to find confidence intervals or test hypotheses; they involve the "error sum-of-squares" $S^2 = \frac{1}{n-2} \sum (Y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2$.

What bothered me is that the slope-intercept form (1) seems unnatural. We were working through an example relating the weight $x$ of various models of cars to their fuel economy $y$ in miles per gallon (mpg), and when we computed the estimated $y$-intercept $\hat{\beta}_0$, a student asked me the meaning of this number. I had to say "it's the mpg that the model predicts for a car with zero weight", which is a pretty silly quantity to consider unless extrapolating is your hobby. Moreover, there's the issue that $\hat{\beta}_0, \hat{\beta}_1$ are very dependent. You could find confidence intervals for each of them, but looking at the endpoints of those intervals together would not give you a reasonable picture of the most "extreme" line consistent with the model.

So it seemed to me that a "point-slope" form would be more appropriate: $$Y_i = \alpha + \beta_1(x_i - \bar{x}) + \epsilon_i \tag{2}$$ After a little searching, I guess this is called "centering", but I didn't find sources that work with this version in detail. So my first question is: is there a standard name and/or notation for the parameter I am calling $\alpha$? It represents the response predicted by the model for the average predictor value $\bar{x}$ (in our example, the predicted fuel economy of an average-weight car).

Now the obvious estimator for $\alpha$ would be $\hat{\alpha} = \bar{Y} = \alpha + \bar{\epsilon}$. I'd like to work out the corresponding pivotal quantity and its distribution.

• First, it appears to me that $\bar{Y}, \hat{\beta}_1, S^2$ are mutually independent, though I didn't actually prove it as it seems tedious. But is this true?

• Next, it seems clear that $\bar{Y} \sim N(\alpha, \sigma^2/n)$, so it's an unbiased estimator of $\alpha$, but of course $\sigma^2$ is unknown. However, we have the pivot $\frac{n-2}{\sigma^2} S^2 \sim \chi^2(n-2)$. So if $S^2$ is indeed independent of $\bar{Y}$, we ought to have $$\frac{\bar{Y} - \alpha}{S/\sqrt{n}} \sim t(n-2). \tag{3}$$ This would allow us to find a confidence interval for $\alpha$. So have I got this right? I was hoping I would find this formula in a book somewhere to confirm it, but I haven't found it yet.

References that discuss this "centered" or "point-slope" version would be welcome, especially if they are undergraduate-level.

Answer 1

Most of the time in linear models, the intercept is not meant to be interpreted, instead it is to get rid of potentially biased estimates. If we omit the intercept then we force our regression line to past through the origin meaning our slope coefficients - which are the ones we actually care about - will likely biased. Further, if we don't include the intercept we can have potentially negative $R^2$ values and the equality $TSS=ESS+RSS$ may not hold. There are a few situations when forcing the line to go through the origin is reasonable (the CAPM model is a typical example), although these are pretty much all due to some underlying theory stating that it should pass through the origin. That being said, there are some occasions where the intercept does have a reasonable interpretation, but these are just accidental when the regressor could reasonably be 0.

Additionally, I believe the reason for many textbooks not going into too much detail about the 'centred' form could be because a similar form can be recovered from partitioned regression using the Frisch-Waugh method. Hopefully, this is the sort of thing you were looking for when talking about centering the variables.

Using matrix notation can make things easier to show, although a more scalar approach can be recovered quickly either way.

Suppose we have a linear model $y=X\beta +\epsilon$. If we define the column vector $\iota =(1,1,\dots , 1)'$ and a vector space $V=\text{span}(\iota)$ then the projection matrix onto the space orthogonal to $V$ is given by $N=I_n-\iota (\iota' \iota)^{-1} \iota'$. The matrix $N$ has the property that it gives deviations from the mean of a given vector. For example, $$Ny=y-\iota (\iota'\iota)^{-1} \iota'y=y-\iota [n^{-1}\sum y]=y-\iota \bar{y}$$ Further, for any constant vector $c=(c,\dots,c)'$ it has the property of annihilating this vector, so $Nc=0$. Lastly, since we assume $\mathbb{E}[\epsilon]=0$, we have that $N\epsilon =\epsilon$. Using this matrix, we can extract all variation in the model due to the constant to obtain the model $$Ny=NX\beta +N\epsilon=NX\beta+\epsilon$$. The new model provides the OLS estimator $$b=(X'NX)^{-1}X'Ny$$. To make it explicit, if $X$ is a matrix of a single regressor our model $Ny=NX\beta +\epsilon$ becomes $$y_i-\bar{y}=\beta_1 (x_i-\bar{x})+\epsilon$$ where the $\beta_1$ is equal to the one from the original model since $$b=(X'NX)^{-1}X'Ny=([NX]'NX)^{-1}[NX]'Ny=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2}$$. Now we see that the interpretation of $\beta_1$ can loosely be given as the deviation in $y$ from its mean due to a deviation in $x$ from its mean. Further, our t-test in this new model is exactly the same as in the original model since the variance of $y_i-\bar{y}$ is equal to that of $y_i$ (the error variance has not changed from $\sigma^2$). This method is likely why some textbooks don't go into detail (or even cover) variables which have been centered; the OLS estimates of the slope coefficients themselves already represent a measure of how deviations from the average of $x$ correlates with deviations of $y$ about its mean and the intercept is often best left not interpreted. I hope this is the sort of thing you were looking for.

Answer 2

The centred model is useful, but it has its own interpretational complications

Firstly, you're correct to observe that you need to be careful of the interpretation of the intercept term in regression analysis. There is a general principle in regression analysis that you shouldn't "extrapolate beyond the data range", so you will often find that the natural mathematical interpretation of the intercept term is actually not a sensible interpretation. (The example you give is a common type of case where an explanatory variable must be non-negative, yet the conditioning that yields the intercept term occurs when the explanatory term is zero.) The alternative form you propose is indeed useful for interpretational reasons, and it is usually called the centred model.

Whilst the centred form is useful in some ways, it is arguably more complicated to interpret, due to the fact that you now have a model parameter that varies with $n$. To see this, let's write out the centred model form:

$$Y_i = \underbrace{(\beta_0 + \beta_1 \bar{x}_n)}_{\alpha_n} + \beta_1 (x_i-\bar{x}_n) + \varepsilon_i \quad \quad \quad \quad \quad \varepsilon_1,...,\varepsilon_n \sim \text{IID N}(0, \sigma^2).$$

Comparing this model to the standard form we have $\alpha_n \equiv \beta_0 + \beta_1 \bar{x}_n$ and since $\beta_0$ and $\beta_1$ are fixed parameters in the model, the intercept term in this alternative form is implicitly dependent on $n$ through its dependence on the sample mean $\bar{x}_n$. That is, strictly speaking, the centred model actually has a series of intercept parameters $\alpha_1,\alpha_2,\alpha_3,...$ rather than a single parameter $\alpha$.

You can of course estimate the parameters in the alternative centred form of the model using OLS estimation and build up the corresponding theory of how these estimators behaviour. However, this arguably leads you to a harder interpretational problem because you are no longer estimating a fixed intercept parameter --- each estimator $\hat{\alpha}_n$ is estimating a different intercept parameter $\alpha_n$. In any case, the OLS estimator for the coefficients in the centred model are given by:

$$\hat{\alpha} = \bar{y}_n \quad \quad \quad \quad \quad \hat{\beta} = \frac{\sum (x_i-\bar{x}_n)(y_i-\bar{y}_n)}{\sum (x_i-\bar{x}_n)^2}.$$

If we denote the centred design matrix $\mathbf{x}_*$ then the resulting OLS estimator for the centred regression is:

$$\begin{align} \begin{bmatrix} \hat{\alpha} \\ \hat{\beta}_1 \end{bmatrix} &= (\mathbf{x}_*^\text{T} \mathbf{x}_*)^{-1} \mathbf{x}_*^\text{T} \mathbf{Y}, \\[6pt] \end{align}$$

which gives the resulting distribution:

$$\begin{align} \begin{bmatrix} \hat{\alpha} \\ \hat{\beta}_1 \end{bmatrix} &\sim \text{N}(\mathbf{0}, \sigma^2 (\mathbf{x}_*^\text{T} \mathbf{x}_*)^{-1}). \\[6pt] \end{align}$$

Unlike in the non-centred model (where the elements of the OLS estimator are usually correlated), the elements of the estimator are independent in this case, due to the fact that the centering in the design matrix gives:

$$\mathbf{x}_*^\text{T} \mathbf{x}_* = \begin{bmatrix} n & 0 \\ 0 & \sum (x_i - \bar{x}_n)^2 \end{bmatrix}.$$

Since you have a strong mathematics background, you might be interested to read some more about this model form and its geometric interpretation in O'Neill (2019). Centering the explanatory variables allows us to see some useful geometric properties of the regression model that are quite useful for interpretive purposes.