5
$\begingroup$

I have a multiple linear regression problem $y=X\beta+\epsilon$. The number of observations $m$ is large, so by the time the data gets to me it's been summarized into:

  • $m$
  • $X^TX$
  • $X^Ty$
  • $y^Ty$
  • $\sum_{i=1}^m{y_i}$

The above list appears sufficient to compute the OLS estimate. But can it be used to compute a regularized estimate of some sort (e.g. ridge regression)? If not, can the list be augmented in the same spirit to enable regularization?

$\endgroup$
5
$\begingroup$

It would be nice to have in addition the covariance matrix of the residuals $\hat{\Sigma}$ to draw common inferences about the significance of estimated parameters, or if you are sure it is homoscedastic, then just $\hat{\sigma}^2$.

As for the regularizations of generalised least squares type (probably including instrumental variables estimators) the answer will be no. You need the original data matrices (though if you are supplied by $X^T \Omega^{-1}X$ and $X^T\Omega^{-1}y$ you may do GLS, but you loose the control for the choice of $\Omega$ anyway).

For general non-linear lasso regularization it would be even more complicated. Luckily it may be approximated by ridge regression (see p. 273 in the reference) of a special type.

Regarding ordinary ridge regression it is sufficient, since all you need to do in this case is just to add elements to the diagonal $X^TX+\delta I$, where $I$ is an identity matrix. Thus in this particular case it works well.

$\endgroup$
  • 1
    $\begingroup$ Thanks a lot for the insights. It looks like ordinary ridge regression is the way to go for my application. I've posted a follow-on question about choosing a good value for $\delta$ (stats.stackexchange.com/questions/9467/…) $\endgroup$ – NPE Apr 12 '11 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.