Prove that the following is the least squares estimator for $\beta$

Question

Given is that all the assumptions for a linear regression $y_i=x_i'\beta + \epsilon_i$ hold for $i=1,\dots, n$. For the (n+1)st observation we can write: $y_{n+1}=x'_{n+1} \beta + \nu + \epsilon_{n+1} $Then prove that the Least Squares Estimator (LSE) for $\beta$ for the full sample is $b \ (=(X'X)^{-1}X'y)$.

---

I can easily understand that for the first $n$ observations we get the 'standard' LSE. For the last observation, I tried starting from scratch again by minimizing the sum of squared residuals.

What I got is the following: $\frac{\partial (e'e)}{\partial b}=-2x_{n+1}(y_{n+1} -x'_{n+1} b - \nu)=0$. I am not sure though if this helps me, as it doesn't seem to lead to the desired result. Could anyone please help me?

P.S: I prefer hints only so that I can learn something from it.

Answer 1

The regression specification here is actually

$$y_i =\beta_1x_i + \nu\cdot I_{\{i=n+1\}} +\epsilon_i, \;\;i=1,...,n+1$$

which leads us to the OLS estimator

$$\begin{pmatrix} \hat \beta_1\\ \hat v \end{pmatrix} = \frac{1}{\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1} & \sum_{i=1}^{n+1}x_i^2 \end{pmatrix} \begin{pmatrix} \sum_{i=1}^{n+1}x_iy_i \\ y_{n+1} \end{pmatrix} $$

where the last $2\times 1$ vector is $Z'y$. So we obtain the following equation for the OLS estimator of $\beta$:

$$\hat \beta_{1,n+1} = \frac{1}{\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2} \left(1\cdot \sum_{i=1}^{n+1}x_iy_i - x_{n+1}y_{n+1}\right)$$

Note that $$\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2 = \sum\limits_{i=1}^{n} x_i^2$$

and

$$1\cdot \sum_{i=1}^{n+1}x_iy_i - x_{n+1}y_{n+1} = \sum_{i=1}^{n}x_iy_i$$

So

$$\hat \beta_{1,n+1} =\frac {\sum_{i=1}^{n}x_iy_i}{\sum\limits_{i=1}^{n} x_i^2} = \hat \beta_{1,n}$$

i.e. it is the same as the estimator we obtain from the sample of size $n$.

This generalizes to the case of many regressors, and of intermediate observations, and leads to the following general conclusion: in OLS estimation, including a dummy variable that takes the value $1$ for only one observation, and is zero in all others, effectively removes the observation from the sample, as regards the estimation of the coefficients of the other regressors. This is the classic way to treat "outliers" or an observation where "something extraordinary has happened", and one feels that this observation would misleadingly affect the estimation of the parameters, estimation which attempts to capture some "long-term", "structural" relation between the dependent variable and the regressors.

Answer 2

@Alecos, I think I kind of understand your hint now, so here is my attempt (too long for the comments).

Edit made after comments:

For simplicity assume that $\beta=\begin{pmatrix} \beta_1 \end{pmatrix}$. Let $\alpha=\begin{pmatrix} \beta_1\\ \nu \end{pmatrix}$. Also, let $Z=\begin{pmatrix} X & e_{n+1} \end{pmatrix}$ where $e_{n+1}$ is an $(n+1) \times 1$ vector that consists of zeros everywhere, except for in the (n+1)st row.

Then $Z'Z=\begin{pmatrix} X'X & X'e_{n+1} \\ e_{n+1}'X & e_{n+1}'e_{n+1} \end{pmatrix}$. So $(Z'Z)^{-1}= \frac{1}{\operatorname{det}(Z'Z)}\begin{pmatrix} e_{n+1}'e_{n+1} & -e_{n+1}'X \\ -X'e_{n+1} & X'X \end{pmatrix} = \frac{1}{\operatorname{det}(Z'Z)}\begin{pmatrix} 1 & -x_{n+1}\\ -x_{n+1}'& X'X\end{pmatrix}$.

One more edit: $(Z'Z)^{-1}=\frac{1}{\sum\limits_{i=1}^{n+1} (x_i)^2 - x_{n+1}^2} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1}' & X'X \end{pmatrix} = (X'X)^{-1} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1}' & X'X \end{pmatrix}$

And then somehow I need to show that $(Z'Z)^{-1}Z'y=(X'X)^{-1}X'y$? I'm not sure though how...