In other words, is it feasible any of the various penalized regression techniques (such as ridge regression, lasso, and elasticnet) could completely miss the optimal solution for a regression model due to poorly chosen initial values for the model parameters?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ For what it's worth, I'm not sure greedy algorithms are necessarily sensitive to initial conditions or always miss optimal solutions. My impression is that some min-spanning-tree algorithms and Huffman coding are both greedy and optimal. Still an interesting question though! $\endgroup$– Matt KrauseCommented Feb 15, 2013 at 22:37
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
My understanding is that initial values (usually taken as zero for non-intercepts) are not that influential for these penalization techniques and models such as the logistic. And 'optimal solution' is not very well defined. You need to tell us more. For these techniques what is usually the toughest decision is that of how much penalization to use.
answered Feb 15, 2013 at 21:53
-
$\begingroup$ Thank you Frank, by optimization I meant the choice of parameter values that minimizes the penalized residual sum of squares. Actually ridge regression has a solution which can be explicitly solved as with OLS. However, with lasso & elastic net, I'm confused how the solutions are found for each choice of penalty values. Is the entire parameter space explored or is a Newton-Raphson algorithm used that takes the shortest route from the initial parameter values to the closest local minimum (possibly ignoring a global minimum)? $\endgroup$– RobertFCommented Feb 16, 2013 at 20:39
-
$\begingroup$ @RobertF everyone and their grandmother has come up with a different way of conducting the optimization, and they are all iterative in nature (like Newton-Raphson, although that involves getting a Hessian and the $L_1$ penalty is not differentiable). For the LASSO, for example, the LARS algorithm can be used to get the trajectory of the parameter values as the penalization term is varied. These methods all go to a local minimum, but if everything is convex then there is only one local optimum, and it is the global optimum. $\endgroup$– guyCommented Feb 16, 2013 at 22:56
$\begingroup$
$\endgroup$
1
If the loss and penalty are convex functions of your parameters as in a linear regression or GLM then no, you won't get stuck in the wrong place (although it could take awhile to get where you want to go). On the other hand, if you are training (say) a multilayer neural net with quadratic loss using backprop and an L2 penalty then you can very easily get stuck because the loss is not convex as a function of the parameters. In these cases, people typically are just hoping to get a good local solution (which might generalize better anyways), not the best solution.
-
$\begingroup$ Ok thanks guy, this tells me I need to be careful which model (decision tree vs linear or GLM) I use. $\endgroup$– RobertFCommented Feb 16, 2013 at 21:02