3
$\begingroup$

Given is that all the assumptions for a linear regression $y_i=x_i'\beta + \epsilon_i$ hold for $i=1,\dots, n$. For the (n+1)st observation we can write: $y_{n+1}=x'_{n+1} \beta + \nu + \epsilon_{n+1} $Then prove that the Least Squares Estimator (LSE) for $\beta$ for the full sample is $b \ (=(X'X)^{-1}X'y)$.


I can easily understand that for the first $n$ observations we get the 'standard' LSE. For the last observation, I tried starting from scratch again by minimizing the sum of squared residuals.

What I got is the following: $\frac{\partial (e'e)}{\partial b}=-2x_{n+1}(y_{n+1} -x'_{n+1} b - \nu)=0$. I am not sure though if this helps me, as it doesn't seem to lead to the desired result. Could anyone please help me?

P.S: I prefer hints only so that I can learn something from it.

$\endgroup$
  • $\begingroup$ What is $\nu$? Is it random? $\endgroup$ – mpiktas Jan 28 '14 at 20:05
  • $\begingroup$ Treat $\nu$ as an unknown constant term that appears only in the $n+1$ observation (so what is the additional regressor series that corresponds to such a situation?). For the $n+1$ sample size, invert the $2\times 2$ moment matrix (easy), derive the values of the OLS coefficient estimates and manipulate, to see that $x_{n+1}$ does not affect the OLS estimate of $\beta$. I will post the answer after some time. $\endgroup$ – Alecos Papadopoulos Jan 28 '14 at 21:36
  • $\begingroup$ @AlecosPapadopoulos Thanks for your help. Sorry, your hints don't yet make sense to me. What do you mean by 'additional regressor series'? Also, I'm not sure what a moment matrix is (and especially what it is in this case). After knowing that, I'd like to try again and see if I then understand :). Thanks again. $\endgroup$ – abc Jan 29 '14 at 14:02
  • $\begingroup$ @mpiktas No, it is not random. It is just an additional constant term that only holds for the last observation. $\endgroup$ – abc Jan 29 '14 at 14:04
  • $\begingroup$ @AlecosPapadopoulos I've been thinking a little longer about your hint and have posted my attempt as an answer below. Could you please look at it? Thank you in advance :). $\endgroup$ – abc Jan 29 '14 at 14:32
3
$\begingroup$

The regression specification here is actually

$$y_i =\beta_1x_i + \nu\cdot I_{\{i=n+1\}} +\epsilon_i, \;\;i=1,...,n+1$$

which leads us to the OLS estimator

$$\begin{pmatrix} \hat \beta_1\\ \hat v \end{pmatrix} = \frac{1}{\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1} & \sum_{i=1}^{n+1}x_i^2 \end{pmatrix} \begin{pmatrix} \sum_{i=1}^{n+1}x_iy_i \\ y_{n+1} \end{pmatrix} $$

where the last $2\times 1$ vector is $Z'y$. So we obtain the following equation for the OLS estimator of $\beta$:

$$\hat \beta_{1,n+1} = \frac{1}{\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2} \left(1\cdot \sum_{i=1}^{n+1}x_iy_i - x_{n+1}y_{n+1}\right)$$

Note that $$\sum\limits_{i=1}^{n+1} x_i^2 - x_{n+1}^2 = \sum\limits_{i=1}^{n} x_i^2$$

and

$$1\cdot \sum_{i=1}^{n+1}x_iy_i - x_{n+1}y_{n+1} = \sum_{i=1}^{n}x_iy_i$$

So

$$\hat \beta_{1,n+1} =\frac {\sum_{i=1}^{n}x_iy_i}{\sum\limits_{i=1}^{n} x_i^2} = \hat \beta_{1,n}$$

i.e. it is the same as the estimator we obtain from the sample of size $n$.

This generalizes to the case of many regressors, and of intermediate observations, and leads to the following general conclusion: in OLS estimation, including a dummy variable that takes the value $1$ for only one observation, and is zero in all others, effectively removes the observation from the sample, as regards the estimation of the coefficients of the other regressors. This is the classic way to treat "outliers" or an observation where "something extraordinary has happened", and one feels that this observation would misleadingly affect the estimation of the parameters, estimation which attempts to capture some "long-term", "structural" relation between the dependent variable and the regressors.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ah, I was close ;). Thanks a lot @Alecos. I really appreciate all your help! Enjoy your evening. $\endgroup$ – abc Jan 29 '14 at 17:26
  • $\begingroup$ Indeed you were. If you feel adventurous, try to prove it for the case of many $\beta$'s! $\endgroup$ – Alecos Papadopoulos Jan 29 '14 at 17:29
0
$\begingroup$

@Alecos, I think I kind of understand your hint now, so here is my attempt (too long for the comments).

Edit made after comments:

For simplicity assume that $\beta=\begin{pmatrix} \beta_1 \end{pmatrix}$. Let $\alpha=\begin{pmatrix} \beta_1\\ \nu \end{pmatrix}$. Also, let $Z=\begin{pmatrix} X & e_{n+1} \end{pmatrix}$ where $e_{n+1}$ is an $(n+1) \times 1$ vector that consists of zeros everywhere, except for in the (n+1)st row.

Then $Z'Z=\begin{pmatrix} X'X & X'e_{n+1} \\ e_{n+1}'X & e_{n+1}'e_{n+1} \end{pmatrix}$. So $(Z'Z)^{-1}= \frac{1}{\operatorname{det}(Z'Z)}\begin{pmatrix} e_{n+1}'e_{n+1} & -e_{n+1}'X \\ -X'e_{n+1} & X'X \end{pmatrix} = \frac{1}{\operatorname{det}(Z'Z)}\begin{pmatrix} 1 & -x_{n+1}\\ -x_{n+1}'& X'X\end{pmatrix}$.

One more edit: $(Z'Z)^{-1}=\frac{1}{\sum\limits_{i=1}^{n+1} (x_i)^2 - x_{n+1}^2} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1}' & X'X \end{pmatrix} = (X'X)^{-1} \begin{pmatrix} 1 & -x_{n+1} \\ -x_{n+1}' & X'X \end{pmatrix}$

And then somehow I need to show that $(Z'Z)^{-1}Z'y=(X'X)^{-1}X'y$? I'm not sure though how...

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You 're close. The $\nu$ appears ONLY in the $n+1$ observation, NOT in the previous ones. So the regressor series corresponding to it is NOT a series of ones, but a series of zeros with only one $1$ in the $n+1$ obs. Before working the case with many $\beta$s (which needs some familiarity with manipulating block matrices), just work the scenario with only one $\beta$ (not even a "usual" constant term" -and this special constant term. That is what I meant by "inverting the $2\times 2$ matrix - that you have only two regressors. $\endgroup$ – Alecos Papadopoulos Jan 29 '14 at 15:09
  • $\begingroup$ @AlecosPapadopoulos Thanks again for your feedback :)! I'll give it a try and then edit in my new attempt. $\endgroup$ – abc Jan 29 '14 at 15:18
  • $\begingroup$ @AlecosPapadopoulos I have edited my answer. Still though, I cannot get to the desired result. Could you show me how I should continue from here and/or what I should do differently? Thanks again. $\endgroup$ – abc Jan 29 '14 at 15:42
  • $\begingroup$ $X'X$ is now one-dimensional. Calculate explicitly the $det$ and then calculate explicitly the OLS estimator $(Z'Z)^{-1}Z'y$ (using perhaps sum-symbols rather than vector products). You will see that eventually the $x_{n+1}$ and the $y_{n+1}$ cancel out, for the determination of $\hat \beta_1$, if you insert the obtained equation for $\hat \nu$ in the equation for $\hat \beta_1$. Now I have revealed everything! $\endgroup$ – Alecos Papadopoulos Jan 29 '14 at 15:48
  • $\begingroup$ @AlecosPapadopoulos I'm sorry, I don't see yet how to do that :(. Could you please post it as an answer? Thanks a lot for all your effort. $\endgroup$ – abc Jan 29 '14 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.