I have seen a method whereby instead of trying to estimate the ridge parameter (k) directly from the data (using one of the many many ridge parameter estimators in the literature) you solve for it iteratively.
The method is simple enough: You simply increase k (in suitably small steps) until the condition number is reduced blow 10.
At first blush this seems like quite a nice solution to me but I've never seen a Ridge Regression paper/book that uses it.
Update OK this is basically the method suggested by Marquardt "Generalized inverses, Ridge Regression, Biased Linear Estimation and Non-linear Estimation" the only difference being he used VIF's to measure the MC while this method uses the condition number. McDonald and Galrneau "A Monte-Carlo Evaluation of some Ridge-Type Estimators" note that this method is may not be appropriate for all data sets as it does not include the y values (observations). I still have not found a paper where the Marquardt method is tested against other estimators for the ridge parameter does anybody know of such a paper?
Is this method theoretically sound though? Even if (as I suspect) it isn't does it really matter for the average practitioner who just want to produce more stable estimates of their Beta's (the weights in the regression) rather than having them "blow up" to grossly unrealistic values when they experience severe MC?
Truly I would like to find a better method than this ideally with a solid theoretical underpinning but its hard to see from a practical view point it can be improved upon?