2
$\begingroup$

Consider a linear regression

$$ y_i = A_i' \theta + B_i' \psi + \epsilon_i $$

There's a "trick" to find the parameters $\theta$ and $\psi$ in a stepwise fashion.

First minimizing squared error wrt. $\theta$: $$\hat{\theta}(\psi) = (A^t A)^{-1} A^t (y - B\psi) $$

Plugging back in, the residuals are $$ \epsilon = ( I - A(A^t A)^{-1} A^t ) (y - B \psi)$$ so the idea is that to find $\hat{\psi}$ by residualizing both $y$ and $B$ with respect to $A$.

Let $y_{\perp A} = (I - A(A^t A)^{-1} A^t) y $ and $B_{\perp A} = (I - A(A^t A)^{-1} A^t) B$. Then you just do $$ \hat{\psi} = (B_{\perp A}^t B_{\perp A})^{-1} B_{\perp A}^t y_{\perp A}$$

My questions are:

  • What is this called? I've tried searching for "stepwise regression", but that's definitely not the right term
  • Is it possible to extend this argument to a case when the dependence on $\psi$ is non-linear? I'm currently fitting the non-linear model to the residuals of the linear one, but I'm not sure if it's a valid thing to do

Many thanks

$\endgroup$

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.