# Automatically fixing ill-conditioning or collinearity

I'm backtesting a regression model, which entails running it on a bunch of bootstrap samples of a "rewound" version of our data set. Unfortunately, in some of these resamplings, I end up getting some "coincidental" dependencies between covariates that result in a covariate matrix that's either rank-deficient or ill-conditioned (on that specific bootstrap sample; the full covariate matrix is not ill-conditioned). Obviously this causes difficulties for inference.

Is there a standard way of solving this automatically/algorithmically? If I were only fitting on one dataset, I'd simply drop one of the features causing the rank deficiency/ill-conditioning, but that's impractical to do for 20 bootstrap samples of 12 different time snapshots.

My default plan is just to switch to penalized regression, but I'm wondering if there's a more principled approach, since I have enough data that I definitely don't need the penalty otherwise.

• I added the variance-covariance tag as this concerns a VCV matrix; I hope you are OK with it. – usεr11852 says Reinstate Monic Apr 17 '15 at 3:28

If you cannot use your original matrix $K$, you might want to use a $K^{\prime}$ that is the closest you can get in terms of Frobenius norm. To do that find the smallest eigenvalues and set them to a higher number positive number. This will give you a new P(S)D matrix that is closer than any other matrix in Frobenius norm, see N.J. Higham 1988 for details.
You can also consider using Tikhonov regularization. In that case you essentially do a ridge regression-like correction. As you original matrix $K$ will become $K^{\prime} = K + \lambda I$, you will add $\lambda$ to all your eigenvalues. While a bit simplistic this choice has a very nice statistical interpretation: you practically say that all your samples are bit ($\lambda$) more noisy than before (and because as you add things along the diagonal no covariances are affected).