# Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

533 questions
Filter by
Sorted by
Tagged with
405 views
+50

### L1 vs L2 stability?

See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/ The instability property of the method of least absolute deviations means that, for ...
43 views

### Simplifying the Matrix Form of the Solution to Ridge Regression

I'm trying to understand how to obtain the solution to an objective function by solving for the parameter vector $\theta$ in ridge regression. I found an example here from Naomi which takes an example ...
12 views

### Why would one want to choose lambda.1se for ridge regression in glmnet?

In R, choosing lambda.1se over lambda.min to get a more parsimonious model is common. This post (and this) also indicated that ...
31 views

### Least Square vs Shrinkage approach of fitting models

What is the difference between the Least Square and Shrinkage approach of fitting models in the context of model selection? In https://www.youtube.com/watch?v=QlyROnAjnEk the author at [0:28] instance ...
168 views

### Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
47 views

### Howe to perform ridge regression only on a subset of the variables

I am trying to code some algorithm that performs ridge-regression with penalty parameter $\lambda$ on all features except for a specific subset. Let $\mathbf{X}$ be the $n \times p$ matrix for $n$ ...
41 views

### Difference between L1 and L2 Regularization (in Lasso and Ridge Regression)

I got a more theoretical question here: I have made some research about the L2 (Ridge) and L1 (Lasso) regularizations. I know the formula, and understand the aim of those two different procedures. The ...
22 views

### Regression with pytorch with inferior results to ridge regression

I am trying to use a simple 3 layer neural net to predict a scaler output given an input of dimension 430. For my network, I use 2 layers of dimensions 600 and 80 and I use leakyReLU non-linearities. ...
52 views

### Why is linear regression not a stable algorithm?

In the paper Stability and Generalization the author defines the stability of a learning algorithm, which intuitively means that changing one sample in the sample set does not affect the outcome much. ...
7 views

### Matlab - Financial Modeling, Linear Regression with Prior

Am trying to implement this equation from the book Doing Data Science Straight Talk from the frontline, In chapter 6, page 161, equation below: From what i can tell it is pretty much an enchanced ...
57 views

### Feature Selection for Ridge Regression

There is a closed-form equation for computing the optimal weight matrix $\mathbf{W}_{\text{optimal}}$ according to the feature matrix $\mathbf{\Phi}$ and the target matrix $\mathbf{Z}$ for ridge ...
16 views

### Can we exclude non-significant variables (p>0.05) in ridge regression?

As far as I know, variables with p>0.05 are non-significant for the regression model. I found similar questions here about p-value like: What is the meaning of p values and t values in statistical ...
33 views

### When to use LSTM vs Lasso/Ridge Regression vs ARIMA?

I have a set of N time series and want to make predictions about the future values of these N elementary time signals. From a first rough analysis, I can say that at a given moment in time, the N ...
4k views

51 views

### Results of cv.glmnet in R versus RidgeCV in scikit-learn

I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R). First a few things to note: ...
12 views

### Getting different values for MSE using anova(lm(y~.)) and mean(residuals(fit)^2)

Using this dataset of gas mileage for different cars I've been asked to run a ridge regression using $\frac{p*\sigma^2}{\beta'\beta}$ as the k-value. I've been told $\sigma^2 = MSE$ $p =$ ...
849 views

### How to decide whether to use Ridge Regression/LASSO/Elastic Net or Random Forest for Feature Selection?

My understanding is rudimentary and high level but it seems like Ridge Regression/LASSO/Elastic Net would be better when the data is linear and Random Forest is better when the data is nonlinear? Also ...
31 views

### Robust regression with M-estimators

I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator: Is it possible/common to combine these with regularization terms, such ...
129 views

### Is group lasso equivalent to ridge regression when there is 1 group

On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso). However in group lasso we have norm ...
26 views

### Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
6k views

### AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
53 views

### Poor performance on Regularized models

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've ...
40 views

994 views

### Rationale behind shrinking regression coefficients in Ridge or LASSO regression

I understand that with Ridge or Lasso regression we are trying to shrink regression coefficients, and we specify the amount of shrinking we need by varying alpha. But I cannot understand the intuition ...
983 views

### The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
239 views

### Ridge regression: penalizing weights corresponding to larger-scale features

In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function. On page 5, in the little section called "...
### glmnet package: “mgaussian” vs “gaussian” for $\alpha = 0$
In multiresponse Gaussian family the objective function when $\alpha = 0$: \begin{align} \frac{1}{2n}||Y-XB||_F^2 + \frac{\lambda}{2}||B||_F^2. \end{align} This can also mathematically solved as \...