I'm backtesting a regression model, which entails running it on a bunch of bootstrap samples of a "rewound" version of our data set. Unfortunately, in some of these resamplings, I end up getting some "coincidental" dependencies between covariates that result in a covariate matrix that's either rank-deficient or ill-conditioned (on that specific bootstrap sample; the full covariate matrix is not ill-conditioned). Obviously this causes difficulties for inference.
Is there a standard way of solving this automatically/algorithmically? If I were only fitting on one dataset, I'd simply drop one of the features causing the rank deficiency/ill-conditioning, but that's impractical to do for 20 bootstrap samples of 12 different time snapshots.
My default plan is just to switch to penalized regression, but I'm wondering if there's a more principled approach, since I have enough data that I definitely don't need the penalty otherwise.