4
$\begingroup$

I have seen a method whereby instead of trying to estimate the ridge parameter (k) directly from the data (using one of the many many ridge parameter estimators in the literature) you solve for it iteratively.

The method is simple enough: You simply increase k (in suitably small steps) until the condition number is reduced blow 10.

At first blush this seems like quite a nice solution to me but I've never seen a Ridge Regression paper/book that uses it.

Update OK this is basically the method suggested by Marquardt "Generalized inverses, Ridge Regression, Biased Linear Estimation and Non-linear Estimation" the only difference being he used VIF's to measure the MC while this method uses the condition number. McDonald and Galrneau "A Monte-Carlo Evaluation of some Ridge-Type Estimators" note that this method is may not be appropriate for all data sets as it does not include the y values (observations). I still have not found a paper where the Marquardt method is tested against other estimators for the ridge parameter does anybody know of such a paper?

Is this method theoretically sound though? Even if (as I suspect) it isn't does it really matter for the average practitioner who just want to produce more stable estimates of their Beta's (the weights in the regression) rather than having them "blow up" to grossly unrealistic values when they experience severe MC?

Truly I would like to find a better method than this ideally with a solid theoretical underpinning but its hard to see from a practical view point it can be improved upon?

Improve this question
$\endgroup$
2
  • $\begingroup$ What's VIF? What's MC? $\endgroup$ – Memming Oct 12 '13 at 14:07
  • $\begingroup$ @Memming - VIF = Variance Inflation Factor, MC is multicollinearity. - Hugh - please don't use acronyms unless they are really, really standard and widely known, like "BIC" or "GLM". $\endgroup$ – jbowman Oct 12 '13 at 15:33
1
$\begingroup$

Ridge regression smoothing (Tikhonov factor, shrinkage, or other synonyms) can be chosen adaptively to minimize the error of a parameter(s) of interest. That parameter need not be directly related to the parameters of the model, it is sufficient that its error can be defined by error propagation defined in terms of the parameters of the model. This is usually applied for ill-posed integrals, for example, the total area under curve from $t=${0,Infinity} extrapolated from incomplete range data.

Note, this answer has nothing whatever to do with goodness-of-fit, it is how to perform an inverse problem solution. That is, it 1) assigns a reason, goal, or target for doing regression, 2) extracts only that optimized information from the model and data, and 3) yields a goodness defined in terms of the regression target and not goodness-of-esthetically-fitting the model to the data. Example, read appendix in this. And no, it isn't perfect either, just better at finding answers.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ A common approach in "classical" regularization applications (typically inversion of physics-based forward models, e.g. tomography) is the "L curve", which tries to balance the data-misfit and regularization residual norms. Modern approaches (e.g. statistics/machine learning) typically choose the regularization strength based on some combination of prior information and/or cross-validation. $\endgroup$ – GeoMatt22 Sep 14 '16 at 6:08
  • 1
    $\begingroup$ @GeoMatt Thanks for the inclusive comment, it is worth mentioning. From my point, the L curve method is heuristic. Prior information is Bayesian reasoning, to which I counter that if one knows so much about the answer, why bother to solve the problem at all? Post hoc analysis of preconditions in less ambiguous than a priori reasoning, where the latter is prejudicial and risks confirmation bias. Cross-validation may afford some opportunity for establishing regression goals beyond simple goodness-of-fit, and may be worth follow-up. $\endgroup$ – Carl Sep 15 '16 at 16:26
  • $\begingroup$ Carl, I was just putting some context for what methods are commonly used for setting the regularization strength - no judgement intended as to when they are appropriate. As for "if you know so much about the answer, why bother ...", an easy counter-example is recursive Bayesian estimation in data assimilation: For example, the Kalman filter update can be interpreted as an instance of Tikhonov regularization. $\endgroup$ – GeoMatt22 Sep 15 '16 at 16:36
  • $\begingroup$ @GeoMatt I understood that and thank-you for your comments. They were taken in the spirit of a compendium of application, by "you" I was not intending a personal slight and apologize and admittedly, "if one were to know" would have been better. Deducing veracity of a Bayesian condition suffers ambiguity not experienced using post hoc analysis $\endgroup$ – Carl Sep 15 '16 at 16:59
  • $\begingroup$ no offense taken, certainly! My intent was only to give an example where Bayesian reasoning is essential. Real-time data assimilation problems such as SLAM are very different from the classic "regression on pre-gathered experimental data" type of statistics problem. $\endgroup$ – GeoMatt22 Sep 15 '16 at 17:08
1
$\begingroup$

I like to think of this as optimizing effective AIC. See the following for a presentation on this topic: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/iscb98.pdf

Improve this answer
$\endgroup$
1
  • $\begingroup$ What a strange comment when the long answer is in the link I provided. $\endgroup$ – Frank Harrell Aug 13 '17 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.