1
$\begingroup$

Consider a data generating mechanism of $Y_i = \mu(X_i)+ \epsilon_i$, $\epsilon_i \sim N(0,1)$. $\mu(X)$ is a nonlinear function. Suppose $X$ is also a random variable with $E[X]=0$ and $Var[X] = \sigma_X$.

We fit a model $Y_i = X_i \beta$ using the Least squares method. Let $$\beta^{*} = argmin_{\beta} E(\mu(X_i) - X_i \beta)^2.$$

What is $Cov(\mu(X), X\beta^{*})$? Is it true that $Cov(\mu(X), X\beta^{*})=Var(X\beta^{*})$?

The context of this question is that I came across this expression when I was reading the following lecture note (pp.7, the note on the bottom).

https://web.stanford.edu/~swager/stats361.pdf

It says that $X\beta^{*}$ is the projection of $\mu(X)$ onto the linear span of the features $X$ but I could not figure out what this means.

Any help would be highly appreciated!

$\endgroup$

1 Answer 1

2
$\begingroup$

A projection of a vector onto a linear space gives the closest point in that linear space, with the useful property that the projection is orthogonal to the 'residual', the difference between the projection and the original vector.

In this context, saying $X\beta^*$ is the projection of $\mu(X)$ onto the linear span of $X$ just means that $X\beta^*$ is the linear combination of the $X$s that minimises the distance to $\mu(X)$, ie, minimise $E[(\mu(X)-X\beta^*)^2]$. So it means exactly what the definition of $X\beta*$ says.

The key consequence of $X\beta$ being the projection is that $\mu(X)-X\beta^*$ is uncorrelated with $X\beta^*$, so $$\mathrm{cov}[X\beta^*,\mu(X)]=\mathrm{cov}[X\beta^*,X\beta^*]+\mathrm{cov}[X\beta^*,\mu(X)-X\beta^*]= \mathrm{var}[X\beta^*]+0$$

The point in the context of your link is that for some purposes it doesn't matter whether $\mu(X)$ is really linear. You can still work with the linear part $X\beta^*$, and the nonlinear $\mu(X)-X\beta^*$ joins $\epsilon$ as part of the error in the linear model

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.