2
$\begingroup$

I was running some LassoCV and RidgeCV and wanted to know whether it is possible for CV MSE functions of lambda for say Ridge regression can have multiple minima.

e.g. multiple values of lambda such that derivatives $dMSE/d\lambda == 0$

(not saddle points or maxima)

typically R or python will show some graph with a minima of MSE vs $\lambda$ and I am just questioning how robust these graphs are if there are multiple minima, is it going to give you the global minimum etc.

EDIT: after reviewing the sub-satsifactory answers here, you can find out more about this issue here: What causes lasso to be unstable for feature selection?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

MSE is comparatively benign wrt. variance uncetainty (random error). Nevertheless, there is such uncertainty on the observed MSE. The observed MSE can therefore have multiple minima (even if that is not the case with the underlying true MSE).

MSE estimates become more uncertain with

  • few test cases (for CV: small absolute number of cases)
  • more unstable models (more complex models)

In practice, most people are not bothered by multiple minima since choosing the least complex model with minimum error (or within a given bound from the observed minimum) is a well working heuristic.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think my question is poorly posed. Assuming you have a sufficiently large non-biased sample from the population - then i'm not asking about shifts in the minima due to variance - more weather there could be some other minima in some other regime of lambda (say change lambda by a factor of 100 and you might find some local minima somewhere) $\endgroup$
    – MaxYarmolinsky
    Commented Oct 7, 2020 at 22:16
2
$\begingroup$

Multiple local minima don't occur for cross-validated MSE because MSE is a convex function. A minimum found in a convex optimization problem (that is uncomplicated and constraint-free) is guaranteed to be a global, not a local, minimum.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ This is more the answer I was looking for, however, your statement could use a little bit of explaining why it is a convex function etc. what do you mean by "complicated" etc. $\endgroup$
    – MaxYarmolinsky
    Commented Oct 7, 2020 at 22:17
  • $\begingroup$ constraints and rules latched onto the core objective function can interfere with an optimizer's path to the global minimum. I guess complicated here means non-convex, which I obviously had no need to even bring up in the first place $\endgroup$
    – develarist
    Commented Oct 8, 2020 at 6:03
  • $\begingroup$ ok, so ridge or lasso obviously minimize mse while subject to constraint of also minimizing betas. Does this make it a complicated or non-complicated function - some sort of rigorous proof is needed $\endgroup$
    – MaxYarmolinsky
    Commented Oct 8, 2020 at 15:30
  • 1
    $\begingroup$ when I irrevocably talked about complicated, it was in no way directed at the penalty term $\endgroup$
    – develarist
    Commented Oct 8, 2020 at 15:31
  • 1
    $\begingroup$ ok, so ridge and lasso are in fact convex... so you are right, sorry $\endgroup$
    – MaxYarmolinsky
    Commented Oct 8, 2020 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.