How to explain the differences between lasso, ridge, and elastic net to a layman? [duplicate]

Question

How to explain the differences between ridge, lasso, and elastic net to a smart layman? Such as including the pros and cons of each method and when they should be used.

Answer 1

For machine learning related problems:

• Ridge: want your weights to be bounded

• Lasso: want your weights to be bounded and potentially stored easily in a computer

• Ridge + Lasso = Elastic Net: want your weights to be bounded and potentially stored easily in a computer

For any more details in terms of the differences you will have to use math.

Of course, some optimization problems have natural $l_1$ or $l_2$ constraints to them, then Lasso or Ridge are a matter of necessity instead of some hand-wavy thing that people do in machine learning, e.g., enforcing sparsity in the solution.

Answer 2

I would start by explaining regularization: however much we want to fine-tune a model to give us a nice tight fit to our training data, we will consciously de-tune it so that it will be "rounder" and better able to model data the model hasn't seen yet. We don't know, but we suspect that irreducible noise has just such a rounding effect, so building a model that is rounder than it needs to be for the training data prepares it to handle a range of real-world variability that a nice, sharp model wouldn't be able to handle. It is rather like "sharpening" a knife to have blunt, but strong, edge rather than a sharp but brittle and breakable one.

But how should we de-tune a model programmatically? There is more than one way to whet an edge! With ridge regression, we use a spherical rule to round the edges. That is, for each parameter we really want to make "predictive" we force ourselves to make many of the other parameters rather small and therefore comparatively dull. Our predictive features will have larger coefficients so they can predict more of the signal or explain more of the fine details in the shape of the classification area. Features with large parameters are emphasized with respect to the others. These are the sharp edges. If a feature's parameter is small, its ability to describe much of the signal is also relatively small. We spread the dullness around so that each edge is still represented. As a result, the dull features average over each other to provide low resolution descriptive/predictive power. You can tell your layman friend that the ridge regression is more like squeezing a large, complex, highly-reticulated sponge so it can fit into a awkward shaped bottle or glass. One or two dimensions are sculpted to adopt the shape of the container, but the others are just left to be average -- a blob.

But that isn't the only way to dull a blade or de-tune a model. With lasso regression, we use a slightly different principle to make sure important features suck up a little bit more of the explanatory power and the dull edges, well, they just evaporate to nothing. It is rather like pruning the features of the sponge that aren't going to fit into the container, thereby given more "room" for the more appropriate features to fit the form. The way we optimize our model on the training data follows a rule that drives the importance of the non-predictive features all of the way to zero, thereby providing a bit more explanatory power to the few remaining important features.

Elastic net is a simple compromise of these methods. We budget some of our predictive power to mold some features with precision while others spread out an average background model, and we budget the rest of our predictive power to form a lower dimensional specific shape, having driven all marginal features right out of the model. Both sponges are in play. We just use more or less of the one that we squeeze and the one that we prune first.

You'd choose ridge regression if you honestly believe all features are significant OR if you are comparing two problems with the same feature set and you'd wish to compare relative parameter values of features in both problem spaces. The research question may be to identify which parameters are "up-regulated" and which are "down-regulated" between the two problems.

You'd choose lasso regression if you don't have a theory to explain the relative importance of features and are looking for a simple low-dimensional model quickly.

Or you'd choose elastic net if you can't decide and you have plenty of computer power to try combinations of both to see which delivers the best overall fit to the training data.