Skip to main content
edited body
Source Link
balaks
  • 441
  • 4
  • 4

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:

  1. Lasso does a sparse selection, while Ridge does not.
  2. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not know which variable gets picked. Elastic-net is a compromise between the two that attempts to shrink and do a sparse selection simultaneously.
  3. Ridge estimators are indifferent to multiplicative scaling of the data. That is, if both X and yY variables are multiplied by constants, the coefficients of the fit do not change, for a given $\lambda$ parameter. However, for Lasso, the fit is not independent of the scaling. In fact, the $\Lambda$$\lambda$ parameter must be scaled up by the multiplier to get the same result. It is more complex for elastic net.
  4. Ridge penalizes the largest $\beta$'s more than it penalizes the smaller ones (as they are squared in the penalty term). Lasso penalizes them more uniformly. This may or may not be important. In a forecasting problem with a powerful predictor, the predictor's effectiveness is shrunk by the Ridge as compared to the Lasso.

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:

  1. Lasso does a sparse selection, while Ridge does not.
  2. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not know which variable gets picked. Elastic-net is a compromise between the two that attempts to shrink and do a sparse selection simultaneously.
  3. Ridge estimators are indifferent to multiplicative scaling of the data. That is, if both X and y variables are multiplied by constants, the coefficients of the fit do not change, for a given $\lambda$ parameter. However, for Lasso, the fit is not independent of the scaling. In fact, the $\Lambda$ parameter must be scaled up by the multiplier to get the same result. It is more complex for elastic net.
  4. Ridge penalizes the largest $\beta$'s more than it penalizes the smaller ones (as they are squared in the penalty term). Lasso penalizes them more uniformly. This may or may not be important. In a forecasting problem with a powerful predictor, the predictor's effectiveness is shrunk by the Ridge as compared to the Lasso.

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:

  1. Lasso does a sparse selection, while Ridge does not.
  2. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not know which variable gets picked. Elastic-net is a compromise between the two that attempts to shrink and do a sparse selection simultaneously.
  3. Ridge estimators are indifferent to multiplicative scaling of the data. That is, if both X and Y variables are multiplied by constants, the coefficients of the fit do not change, for a given $\lambda$ parameter. However, for Lasso, the fit is not independent of the scaling. In fact, the $\lambda$ parameter must be scaled up by the multiplier to get the same result. It is more complex for elastic net.
  4. Ridge penalizes the largest $\beta$'s more than it penalizes the smaller ones (as they are squared in the penalty term). Lasso penalizes them more uniformly. This may or may not be important. In a forecasting problem with a powerful predictor, the predictor's effectiveness is shrunk by the Ridge as compared to the Lasso.
Source Link
balaks
  • 441
  • 4
  • 4

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:

  1. Lasso does a sparse selection, while Ridge does not.
  2. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not know which variable gets picked. Elastic-net is a compromise between the two that attempts to shrink and do a sparse selection simultaneously.
  3. Ridge estimators are indifferent to multiplicative scaling of the data. That is, if both X and y variables are multiplied by constants, the coefficients of the fit do not change, for a given $\lambda$ parameter. However, for Lasso, the fit is not independent of the scaling. In fact, the $\Lambda$ parameter must be scaled up by the multiplier to get the same result. It is more complex for elastic net.
  4. Ridge penalizes the largest $\beta$'s more than it penalizes the smaller ones (as they are squared in the penalty term). Lasso penalizes them more uniformly. This may or may not be important. In a forecasting problem with a powerful predictor, the predictor's effectiveness is shrunk by the Ridge as compared to the Lasso.