3
$\begingroup$

We know that the best predictor of $\beta$ using the least squares criteria for linear regression is $\hat{\beta} = (X^TX)^{-1}X^Ty$ and I can derive this equation by minimizing the squared error in the training sample.

But when we minimize the expected prediction error, we get: $$ \begin{aligned} \frac{\partial EPE(\beta)}{\partial \beta} = &\iint\frac{\partial}{\partial \beta}(x^T\beta - y)^2dxdy = 0\\ \iff &\iint 2(x^T\beta - y)x\, dxdy = 0 \\ \iff &2\iint x^T\beta x - yx\, dxdy = 0 \\ \end{aligned} $$

Now $x^T\beta$ is a scalar, so $x^T\beta x = x(x^T\beta)$. So: $$ \begin{aligned} \iff &\iint x(x^T\beta) - yx\, dxdy = 0 \\ \iff &E(XX^T\beta) - E(YX) = 0 \\ \iff &\beta = (E(XX^T))^{-1}E(YX) \end{aligned} $$

I dont know how this expectation leads to $\hat{\beta}$. It doesn't seem to add up since $XX^T$ is a NxN matrix. What am I missing?

Thanks!


Edit: since it was pointed out that if $X$ is a random variable, the expectation has to be conditioned, we have:

$$ \begin{aligned} \frac{\partial EPE(\beta)}{\partial \beta} = &\iint\frac{\partial}{\partial \beta}(x^T\beta - y)^2 \Pr(x,y) = 0\\ \iff &\iint 2(x^T\beta - y)x \Pr(x,y) = 0 \\ \iff &2\iint [x^T\beta x - yx] \Pr(y|x) \Pr(x) = 0 \\ \iff &E_XE_{Y|X} [X(X^T\beta) - YX] = 0 \\ \iff &E_X(XX^T\beta) - E_XE_{Y|X}(YX) = 0 \\ \iff &\beta = (E_X(XX^T))^{-1}E_XE_{Y|X}(YX) \end{aligned} $$

$\endgroup$
  • 2
    $\begingroup$ $X$ is not random: either it is viewed as constant or else all expectations are conditional on it. $\endgroup$ – whuber Dec 28 '19 at 20:07
  • $\begingroup$ @whuber I added the solution with the conditional expectation. Does that make sense? I am still unsure how to get to $\hat{\beta}$ from there $\endgroup$ – Victor Dec 28 '19 at 23:08
  • $\begingroup$ You haven't done that consistently: for instance, you still take an unconditional expectation with $E_X(XX^\prime).$ Note, too, that linearity of expectation implies $E_{Y\mid X}(YX) = E(Y\mid X)X.$ $\endgroup$ – whuber Dec 29 '19 at 20:42
  • $\begingroup$ I was thinking and I don't believe we need to condition on $X$. We can solve it with the expectation over the joint distribution $\Pr(x,y)$. And then we get $\beta = (E(XX^T))^{-1}E(YX)$. My problem is relating this formula to $\hat{\beta} = (X^TX)^{-1}X^Ty$. Or how come $E(XX^T) = X^TX$ and $E(YX) = X^Ty$. Again these would be expectations over the joint distribution. $\endgroup$ – Victor Dec 29 '19 at 23:30
  • $\begingroup$ Not conditioning on $X$ would result in values that are useless in most applications: think about how little the result would tell you about predicting or estimating the response for any particular set of regressor values. $\endgroup$ – whuber Dec 30 '19 at 14:34
4
$\begingroup$

Indeed, $X$ is not random as @whuber mentioned in the comments. Recall formulation of the Linear Regression problem:

$$ \begin{aligned} y_j =& \beta_1 x_{j1} + ... + \beta_n x_{jn} + \epsilon_j\\ \mathbf{y} =& X \boldsymbol{\beta} + \boldsymbol{\epsilon} \end{aligned} $$

where $\mathbb{E}\epsilon_j=0$. We are given $x_{ij}$ and want to estimate $\theta_j$. Note, that $x_{ij}$ are somewhat not random. Suppose, we are trying to figure out the properties of the device. We pass $x_{ji}$ as input (we know them precisely) and measure the output $y_j$ with some error $\epsilon_j$. So by definition of the problem:

$$ \mathbb{E}y_j = \beta_1 x_{j1} + ... + \beta_n x_{jn} $$

Hence

$$ \begin{aligned} \mathbb{E}(X^TX) = X^TX\\ \mathbb{E}(X^T\mathbf{y}) = X^T\mathbf{y}\\ \end{aligned} $$

Complete derivation may look like that

\begin{aligned} &\iint\frac{\partial}{\partial \beta}(X\hat{\boldsymbol{\beta}} - \mathbf{y})^2dxdy = 0\\ \iff &2\iint X^T(X\hat{\boldsymbol{\beta}} - \mathbf{y})\, dxdy = 0 \\ \iff &2\iint X^T X\hat{\boldsymbol{\beta}} - X^T\mathbf{y}\, dxdy = 0 \\ \iff &E(X^T X\hat{\boldsymbol{\beta}}) - E(X^T\mathbf{y}) = 0 \\ \iff &X^T X\hat{\boldsymbol{\beta}}- X^T\mathbf{y} = 0 \\ \iff &\hat{\boldsymbol{\beta}} = (X^T X)^{-1} X^T\mathbf{y} \\ \end{aligned}

By the way, I am not sure that it actually makes sense to write integrals and take expectations, because we are dealing with constants. Probably it will be better to stick with something like that:

\begin{aligned} \ &(\mathbf{y} - \hat{\mathbf{y}})^T(\mathbf{y} - \hat{\mathbf{y}}) \rightarrow \min\\ \Rightarrow\ &(\mathbf{y} - \hat{\mathbf{y}})^TX = 0\qquad\text{residual}\perp\mathbf{y}\text{'s manifold}\\ \Rightarrow\ &\mathbf{y}^TX - \hat{\boldsymbol{\beta}}^TX^TX = 0\\ \Rightarrow\ &\hat{\boldsymbol{\beta}} = (X^T X)^{-1} X^T\mathbf{y} \end{aligned}


Edit. About taking expectation, it seems that you are right. But probably there is a small mistake connected with vector derivative. It should be:

$$ \begin{aligned} &2\iint (X^T X\hat{\boldsymbol{\beta}} - X^T\mathbf{y})\mathbb{P}(y|x)\mathbb{P}(x) dxdy = 0 \\ \iff &\mathbb{E}_X(X^TX\hat{\boldsymbol{\beta}}) - \mathbb{E}_X\mathbb{E}_{Y|X}(X^T\mathbf{y})=0 \end{aligned} $$

So

$$ \begin{aligned} \mathbb{E}(X^TX\hat{\boldsymbol{\beta}}) =& \mathbb{E}\sum_{i,j,k} x_{ji}x_{jk} \hat{b_k}\\ =& \sum_{i,j,k} \mathbb{E} x_{ji}x_{jk} \hat{b_k}\\ =& \sum_{k} \hat{b_k} \sum_{i,j} \mathbb{E} x_{ji}x_{jk}\\ =& (\mathbb{E} X^TX)\hat{\boldsymbol{\beta}} \end{aligned} $$

That is why

$$ \begin{aligned} &\mathbb{E}_X(X^TX\hat{\boldsymbol{\beta}}) - \mathbb{E}_X\mathbb{E}_{Y|X} (YX) = 0\\ \iff& \hat{\boldsymbol{\beta}} = (E_X(X^TX))^{-1}E_XE_{Y|X}(X^T\mathbf{y}) \end{aligned} $$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, I understand how to get to the $\hat{\beta}$ given a training sample and trying to minimize the squared error. That makes sense to me. What I was trying to do is calculate the EPE for $X, Y$ random variables and $f$ a linear function with shape $f(x) = x^T\beta$. But like @whuber said, this expectation would have to be conditioned on $X$. But now I am not so sure if it makes sense to do that, since the idea of using a linear function presumes that we have some knowledge about the data. $\endgroup$ – Victor Dec 28 '19 at 22:33
  • $\begingroup$ Since you are interested in stuff like that I can recommend reading this question stats.stackexchange.com/questions/204115/… and watching this video youtube.com/watch?v=zrEyxfl2-a8. Both might help you to clarify some concepts. $\endgroup$ – Jesper for President Dec 28 '19 at 22:56
  • $\begingroup$ I have a persuasion to believe, that one need to have some knowledge about the data to apply this method. Even if you decide to utilise nonlinear regression $y_j = f(z_j) + \epsilon_j$ (which in fact reduces to the linear one), then you need to make initial assumptions about $f$. $\endgroup$ – krashkov Dec 28 '19 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.