132

Consider the vector $\vec{x}=(1,\varepsilon)\in\mathbb{R}^2$ where $\varepsilon>0$ is small. The $l_1$ and $l_2$ norms of $\vec{x}$, respectively, are given by $$||\vec{x}||_1 = 1+\varepsilon,\ \ ||\vec{x}||_2^2 = 1+\varepsilon^2$$ Now say that, as part of some regularization procedure, we are going to reduce the magnitude of one of the elements of $\...


100

Since you ask for insights, I'm going to take a fairly intuitive approach rather than a more mathematical tack: Following the concepts in my answer here, we can formulate a ridge regression as a regression with dummy data by adding $p$ (in your formulation) observations, where $y_{n+j}=0$, $x_{j,n+j}=\sqrt{\lambda}$ and $x_{i,n+j}=0$ for $i\neq j$. If you ...


93

With a sparse model, we think of a model where many of the weights are 0. Let us therefore reason about how L1-regularization is more likely to create 0-weights. Consider a model consisting of the weights $(w_1, w_2, \dots, w_m)$. With L1 regularization, you penalize the model by a loss function $L_1(w)$ = $\Sigma_i |w_i|$. With L2-regularization, you ...


81

In an unpenalized regression, you can often get a ridge* in parameter space, where many different values along the ridge all do as well or nearly as well on the least squares criterion. * (at least, it's a ridge in the likelihood function -- they're actually valleys$ in the RSS criterion, but I'll continue to call it a ridge, as this seems to be ...


56

Let's build on what we know, which is that whenever the $n\times p$ model matrix is $X$, the response $n$-vector is $y$, and the parameter $p$-vector is $\beta$, the objective function $$f(\beta) = (y - X\beta)^\prime(y - X\beta)$$ (which is the sum of squares of residuals) is minimized when $\beta$ solves the Normal equations $$(X^\prime X)\beta = X^\...


50

Roughly speaking, there are three different sources of prediction error: the bias of your model the variance of your model unexplainable variance We can't do anything about point 3 (except for attempting to estimate the unexplained variance and incorporating it in our predictive densities and prediction intervals). This leaves us with 1 and 2. If you ...


47

Tikhonov regularizarization is a larger set than ridge regression. Here is my attempt to spell out exactly how they differ. Suppose that for a known matrix $A$ and vector $b$, we wish to find a vector $\mathbf{x}$ such that : $A\mathbf{x}=\mathbf{b}$. The standard approach is ordinary least squares linear regression. However, if no $x$ satisfies the ...


46

1. Which method is preferred? Yes, elastic net is always preferred over lasso & ridge regression because it solves the limitations of both methods, while also including each as special cases. So if the ridge or lasso solution is, indeed, the best, then any good model selection routine will identify that as part of the modeling process. Comments to my ...


44

Short answer: no difference between Primal and Dual - it's only about the way of arriving to the solution. Kernel ridge regression is essentially the same as usual ridge regression, but uses the kernel trick to go non-linear. Linear Regression First of all, a usual Least Squares Linear Regression tries to fit a straight line to the set of data points in ...


44

In The Elements of Statistical Learning book, Hastie et al. provide a very insightful and thorough comparison of these shrinkage techniques. The book is available online (pdf). The comparison is done in section 3.4.3, page 69. The main difference between Lasso and Ridge is the penalty term they use. Ridge uses $L_2$ penalty term which limits the size of the ...


38

If $\lambda \rightarrow \infty$ then our penalty term will be infinite for any $\beta$ other than $\beta = 0$, so that's the one we'll get. There is no other vector that will give us a finite value of the objective function. (Update: Please see Glen_b's answer. This is not the correct historical reason!) This comes from ridge regression's solution in ...


38

Yes. Yes. LASSO is actually an acronym (least absolute shrinkage and selection operator), so it ought to be capitalized, but modern writing is the lexical equivalent of Mad Max. On the other hand, Amoeba writes that even the statisticians who coined the term LASSO now use the lower-case rendering (Hastie, Tibshirani and Wainwright, Statistical Learning with ...


38

You are right to ask this question. In general, when a proper accuracy scoring rule is used (e.g., mean squared prediction error), ridge regression will outperform lasso. Lasso spends some of the information trying to find the "right" predictors and it's not even great at doing that in many cases. Relative performance of the two will depend on the ...


36

How to decide which regularization (L1 or L2) to use? What is your goal? Both can improve model generalization by penalizing coefficients, since features with opposite relationship to the outcome can "offset" each other (a large positive value is counterbalanced by a large negative value). This can arise when there are collinear features. Small changes in ...


35

There is a simple geometric explanation for why the L1 loss function yields the median. Recall that we are working in one dimension, so imagine a number line spreading horizontally. Plot each of the data points on the number line. Put your finger somewhere on the line; your finger will be your current candidate estimate. Suppose you move your finger a ...


35

The Elements of Statistical Learning by Hastie et al. define ridge regression as follows (Section 3.4.1, equation 3.41): $$\hat \beta{}^\mathrm{ridge} = \underset{\beta}{\mathrm{argmin}}\left\{\sum_{i=1}^N(y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^p \beta_j^2\right\},$$ i.e. explicitly exclude the intercept term $\beta_0$ from the ...


35

The question appears to ask for a demonstration that Ridge Regression shrinks coefficient estimates towards zero, using a spectral decomposition. The spectral decomposition can be understood as an easy consequence of the Singular Value Decomposition (SVD). Therefore, this post starts with SVD. It explains it in simple terms and then illustrates it with ...


34

Connection between James–Stein estimator and ridge regression Let $\mathbf y$ be a vector of observation of $\boldsymbol \theta$ of length $m$, ${\mathbf y} \sim N({\boldsymbol \theta}, \sigma^2 I)$, the James-Stein estimator is, $$\widehat{\boldsymbol \theta}_{JS} = \left( 1 - \frac{(m-2) \sigma^2}{\|{\mathbf y}\|^2} \right) {\mathbf y}.$$ In terms of ...


33

In addition to @whuber's comments (*). The book by Hastie et al Statistical learning with Sparsity discusses this. They also uses what is called the $L_0$ "norm" (quotation marks because this is not a norm in the strict mathematical sense (**)), which simply counts the number of nonzero components of a vector. In that sense the $L_0$ norm is used for ...


32

I think there is no single answer to your question - it depends upon many situation, data and what you are trying to do. Some of the modification can be or should be modified to achieve the goal. However the following general discussion can help. Before jumping to into the more advanced methods let's discussion of basic model first: Least Squares (LS) ...


29

+1 on Glen_b's illustration and the stats comments on the Ridge estimator. I would just like to add a purely mathematical (linear algebra) pov on Ridge regression which answers OPs questions 1) and 2). First note that $X'X$ is a $p \times p$ symmetric positive semidefinite matrix - $n$ times the sample covariance matrix. Hence it has the eigen-...


29

This is regarding the variance OLS provides what is called the Best Linear Unbiased Estimator (BLUE). That means that if you take any other unbiased estimator, it is bound to have a higher variance then the OLS solution. So why on earth should we consider anything else than that? Now the trick with regularization, such as the lasso or ridge, is to add some ...


28

The difference you are observing is due to the additional division by the number of observations, N, that GLMNET uses in their objective function and implicit standardization of Y by its sample standard deviation as shown below. $$ \frac{1}{2N}\left\|\frac{y}{s_y}-X\beta\right\|^2_{2}+\lambda\|\beta\|^2_{2}/2 $$ where we use $1/n$ in place of $1/(n-1)$ for ...


28

A natural regularization happens because of the presence of many small components in the theoretical PCA of $x$. These small components are implicitly used to fit the noise using small coefficients. When using minimum norm OLS, you fit the noise with many small independent components and this has a regularizing effect equivalent to Ridge regularization. This ...


28

In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot. In instances where we want to use regression to make inferences about interventions, we want unbiasedness. Not everything to do with data is a prediction problem.


27

It suffices to modify the loss function by adding the penalty. In matrix terms, the initial quadratic loss function becomes $$ (Y - X\beta)^{T}(Y-X\beta) + \lambda \beta^T\beta.$$ Deriving with respect to $\beta$ leads to the normal equation $$ X^{T}Y = \left(X^{T}X + \lambda I\right)\beta $$ which leads to the Ridge estimator.


27

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net: Lasso does a sparse selection, while Ridge does not. When you have highly-correlated variables, Ridge regression shrinks the two coefficients towards one another. Lasso is somewhat indifferent and generally picks one over the other. Depending on the context, one does not ...


25

Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a diagonal matrix with diagonal elements $s_i$. The fitted values of ordinary least squares (OLS) regression are given by $$\hat {\mathbf y}_\mathrm{OLS} = \mathbf X \beta_\mathrm{OLS} = \mathbf X ...


25

Suppose you have two highly correlated predictor variables $x,z$, and suppose both are centered and scaled (to mean zero, variance one). Then the ridge penalty on the parameter vector is $\beta_1^2 + \beta_2^2$ while the lasso penalty term is $ \mid \beta_1 \mid + \mid \beta_2 \mid$. Now, since the model is supposed highly colinear, so that $x$ and $z$ ...


24

This explanation is a summation of muratoa and Yves's comments on D.W.'s answer. Though it is based on calculus, I found it straightforward and easy to understand. Assuming we have $y_1, y_2, ... y_k$ and to want get a new estimate $\beta$ based on them. The smallest loss is obtained when we find $\beta$ which makes the derivative of the loss to zero. L1 ...


Only top voted, non community-wiki answers of a minimum length are eligible