# Tag Info

Accepted

### Is Tikhonov regularization the same as Ridge Regression?

Tikhonov regularizarization is a larger set than ridge regression. Here is my attempt to spell out exactly how they differ. Suppose that for a known matrix $A$ and vector $b$, we wish to find a ...
• 13.2k
Accepted

### Is regression with L1 regularization the same as Lasso, and with L2 regularization the same as ridge regression? And how to write "Lasso"?

Yes. Yes. LASSO is actually an acronym (least absolute shrinkage and selection operator), so it ought to be capitalized, but modern writing is the lexical equivalent of Mad Max. On the other hand, ...
• 91.6k
Accepted

### The proof of shrinking coefficients using ridge regression through "spectral decomposition"

The question appears to ask for a demonstration that Ridge Regression shrinks coefficient estimates towards zero, using a spectral decomposition. The spectral decomposition can be understood as an ...
• 325k
Accepted

### If only prediction is of interest, why use lasso over ridge?

You are right to ask this question. In general, when a proper accuracy scoring rule is used (e.g., mean squared prediction error), ridge regression will outperform lasso. Lasso spends some of the ...
• 93.2k

### Why do we only see $L_1$ and $L_2$ regularization but not other norms?

In addition to @whuber's comments (*). The book by Hastie et al Statistical learning with Sparsity discusses this. They also uses what is called the $L_0$ "norm" (quotation marks because this is not ...
Accepted

### Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

A natural regularization happens because of the presence of many small components in the theoretical PCA of $x$. These small components are implicitly used to fit the noise using small coefficients. ...
• 8,597
Accepted

### What are the implications of scaling the features to xgboost?

XGBoost is not sensitive to monotonic transformations of its features for the same reason that decision trees and random forests are not: the model only needs to pick "cut points" on features to split ...
• 91.6k

### L1 regression estimates median whereas L2 regression estimates mean?

This explanation is a summation of muratoa and Yves's comments on D.W.'s answer. Though it is based on calculus, I found it straightforward and easy to understand. Assuming we have $y_1, y_2, ... y_k$...
• 461

### Why is glmnet ridge regression giving me a different answer than manual calculation?

The difference you are observing is due to the additional division by the number of observations, N, that GLMNET uses in their objective function and implicit standardization of Y by its sample ...
• 511
Accepted

• 105k