# 1. Which methods is preferred?

Yes, elastic net is always preferred over lasso & ridge regression because it solves the limitations of both methods, while also including each as special cases. So if the ridge or lasso solution is, indeed, the best, then any good model selection routine will identify that as part of the modeling process.

Comments to my post have pointed out that the advantages of elastic net are not unqualified. This is, of course, true, but I persist in my belief that the generality of the elastic net regression is still preferable to either L1 or L2 regularization on its own. Specifically, I think that the points of contention between myself and others is directly tied to what assumptions we are willing to make about the modeling process. *In the presence of strong knowledge about the underlying data, some methods will be preferred to others. While this entirely hinges on the judgement of the analyst, that doesn't mean it's incorrect.*

  1. Claim: Prior knowledge may obviate one of the need to use elastic net regression. 

This is somewhat circular. Forgive me if this is somewhat glib, but if you know that LASSO (ridge) is the best solution, then you won't ask yourself how to appropriately model it; you'll just fit a LASSO (ridge) model. I wholeheartedly agree that you should use all available prior information in modeling, but I think there's some unspoken context to OP's question, which is what to do in the general case.

  2. Claim: Modestly large data will not permit discovery of L1 or L2 solutions as preferred, even in cases when the L1 or L2 solution is the true model.

This is also true, but I think it's circular for a similar reason: if you've estimated am optimal solution for $\alpha\not\in \{0,1\},$ then that's the model that the data support. On the one hand, yes, your estimated model is not the true model, but I must wonder how one would know that the true model is $\alpha=1$ (or $\alpha=0$) prior to any model estimation. There might be domains where you have this kind of prior knowledge, but my professional work is not one of them.

  3. Claim: Introducing additional hyperparameters increases the compuational cost of estimating the model.

This is only relevant if you have tight time/computer limitations. GLMNET is the gold-standard algorithm for estimating elastic net solutions. Using it means that you supply some value of alpha, and it uses the path properties of the regularization solution to quickly estimate a family of models for a variety of values of the penalization magnitude, and it can often estimate this family of solutions more quickly than estimating just one solution for a specific value of penalization magnitude. So, yes, using GLMNET does consign you to the domain of using grid-style methods (iterate over some values of $\alpha$ and let GLMNET try a variety of $\lambda$s), but it's pretty fast.

4. Claim: Improved performance of elastic net over LASSO or Ridge Regression is not guaranteed.

This is true, but at the step where one is contemplating which method to use, one will not know which of elastic net, ridge or LASSO is the best. If one reasons that the best solution must be LASSO or ridge regression, then we're in the domain of claim (1). If we're still uncertain which is best, then we can test LASSO, ridge and elastic net solutions, and make a choice of a final model at that point (or, if you're an academic, just write your paper about all 3). This will either place us in the domain of claim (2), where the true model is LASSO/ridge but we did not know so ahead of time, and we accidentally select the wrong model due to uncertainty, or elastic net is actually the best solution.

5. Claim: [Hyperparameter selection without cross-validation is highly biased and error-prone][1].

I hope Scortchi knows me well enough to know that I'm not suggesting that one undertake such an endeavor. Proper model validation is an integral part of any machine learning enterprise. I believe Scortchi's argument is that model validation is also the most expensive step, so one would seek to minimize inefficiencies in that step -- if one of those inefficiencies is needlessly trying $\alpha$ values that are known to be futile, then Scortchi's suggestion is to skip that step. Yes, by all means do that, but we're back to the territory of claim (1) and claim (2).

#2. What's the intuition and math behind elastic net?

I strongly suggest reading the literature on these methods, starting with the original paper on the elastic net. The paper develops the intuition and the math, and is highly readable. Reproducing it here would only be to the detriment of the authors' explanation. But the high-level summary is that the elastic net is a convex sum of ridge and lasso penalties, so the objective function for a Gaussian error model looks like
$$\text{Residual Mean Square Error}+\alpha \text{Ridge Penalty}+(1-\alpha)\text{LASSO Penalty}$$ 

for $\alpha\in[0,1].$

Hui Zou and Trevor Hastie. "[Regularization and variable selection via the elastic net][2]." J. R. Statstic. Soc., vol 67 (2005), Part 2., pp. 301-320.

Richard Hardy points out that this is developed in more detail in Hastie et al. "The Elemets of Statstical Learning" chapters 3 and 18.


  [1]: http://stats.stackexchange.com/questions/137481/how-bad-is-hyperparameter-tuning-outside-cross-validation
  [2]: https://web.stanford.edu/~hastie/Papers/B67.2%20(2005)%20301-320%20Zou%20&%20Hastie.pdf