1
$\begingroup$

I have a multivariate regression model $Y=X\beta ' + \epsilon$. The variables in the $X$ matrix have very different scales and hence the condition number of $X'X$ is huge (order of trillions).

I would like to know if there are problems with parameter estimation due to the high condition number. On one hand, I suspect that if the number is high, the estimates of the $\beta$ are very unstable (because a small change in $X$ could have a large impact on the solution of $X'X\hat{\beta}=X'Y$). On the other hand, I do not think the stability of the solution shall change if I just change the units of the data matrix $X$, because the new estimates should just be multiples of the previous estimates.

Could someone provide advice?

Thanks.

$\endgroup$
  • 1
    $\begingroup$ Such a condition number is meaningless, because the first thing any decent software will do is standardize the columns of $X$. What is the condition number of the matrix your solver is actually working with? $\endgroup$ – whuber Oct 21 '14 at 6:33
2
$\begingroup$

what you are looking for is ridge regression- basically it adds a regularisation term $\alpha \|w\|^2 $, where $w$ is the coefficient vector, to the mean squared error (see eg wiki Tikhonov regularisation). This term penalises large weights (and so rescaling now has some benefit): in particular it penalises solutions with large opposing positive and negative weights

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. So do really different scales of the variables cause (relative) instability of the estimates? $\endgroup$ – DatamineR Oct 20 '14 at 23:57
  • $\begingroup$ The individual scales of the variables are not the source of the large condition number. The condition number is the ratio between the largest and smallest eigenvalues of $X^T X$, which is to do with the correlations between the variables. Your matrix is likely rank deficient, perhaps the smallest eigenvalues are practically zero. $\endgroup$ – purple51 Oct 21 '14 at 0:04
  • $\begingroup$ @purple51 - consider columns of X being independent -what are your eigenvalues? $\endgroup$ – seanv507 Oct 21 '14 at 0:07
  • $\begingroup$ hmm - basically you often want to enforce a "physical" intuition - small changes to the input should have small changes to the output. Using the above scalar $alpha$ regularisation, ideally you would rescale your variables so that a unit change in each variable has a roughly similar sized response- then alpha basically is a cutoff between true signal and (small) measurement noise etc. $\endgroup$ – seanv507 Oct 21 '14 at 0:35
  • $\begingroup$ Yes, you're right, but in I would suspect that in many cases the scaling is less of a problem than multicollinearity if you're doing double precision computation, unless the differences in scale are truly astronomical. But agree about scaling and using some regularization like ridge regression. $\endgroup$ – purple51 Oct 21 '14 at 2:06
2
$\begingroup$

It's a simple fact and easily verified that multiplying one column of X by a scaling factor of say 1,000,000 can dramatically change the condition number of X. Your intuition about the effect of scaling is wrong.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.