Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
746
questions
2
votes
1
answer
33
views
Interpreting the biased coefficients of a ridge or LASSO regression model [duplicate]
In a recent conversation with one of the colleagues I was presented with a view that LASSO/Ridge regularization (trading bias for variance) renders coefficient estimates useless for interpretation, i....
1
vote
1
answer
45
views
Question about retraining a regression model
Exercise. Suppose you train a Ridge model to a regression problem that has a normalized perfomance measure (say K) that attains a value in the interval [0,1], where 0 means that the model is terrible ...
- 125
0
votes
0
answers
26
views
Ridge Estimator in Summation Form
I am trying to derive $\widehat{\beta}$ in summation form from the following:
$$\text{argmin } \sum_{i=1}^{N}(y_i - X_i^{T}\beta)^2 + \lambda \sum_{k=1}^{K}{\beta}_k^2$$
I do not want to resort to ...
- 101
1
vote
1
answer
30
views
glmnet Ridge Regression Plot makes no sense (to me at least)
I have a data set with around 50 variables and I am applying ridge and lasso on this data set. What I´ve noticed is, that the plot for the lambda values does differ from the mean values I get when ...
- 11
3
votes
0
answers
40
views
Geometric intuition for how ridge ($L_2$) regularization helps under multicollinearity
We have some nice posts (1, 2 and likely more) illustrating multicollinearity geometrically. Now, ridge regression ($L_2$ regularization) is known to be a remedy of multicollinearity. What is the ...
- 61.1k
2
votes
0
answers
28
views
If L2-Regularization includes no bias, why do many images show a circle as the constraint region?
I got a little bit (massively, to be honest), confused by the following apparent misconceptions I have learned recently.
Looking for information about L2-Regularization, the following image is one of ...
- 351
3
votes
1
answer
66
views
What does size of coefficients have to do with multicollinearity or overfitting?
In the section on Ridge Regression (source: Elements of Statistical Learning by Hastie, Tibshirani, Friedman) :
When there are many correlated variables in a linear regression model, their ...
- 4,098
0
votes
0
answers
24
views
Why don't Lasso and Ridge Coefficients Correlate in Penalized Linear Regression? [duplicate]
I have fitted Lasso and Ridge regressions on the same training data and having checked the training MSE error seems more-less the same:
...
- 725
0
votes
0
answers
35
views
Variance in Generalized Ridge Regression/Weighted Least Squares
I'm following this collection of papers regarding ridge regression, https://arxiv.org/pdf/1509.09169.pdf , and I ran into this section on the mentioning of generalizing ridge regression.
And when ...
- 469
0
votes
0
answers
24
views
Elastic net can be seen as lasso
Let $y \in \Bbb R^n$, $\Bbb 1$ be an n-vector with all its entries equal to $1$, and $Z \in \Bbb R^{n×p}$ with columns of
unit norm and such that $Z^T \Bbb 1 = 0$. The elastic net is a penalized ...
- 189
3
votes
1
answer
377
views
What would be the solution of Ridge regression, if there is an intercept?
So far I've only seen this solution: $$\beta = (X^TX+\lambda I)^{-1}X^Ty.$$
But I assume this is for the case: $$y=\beta X+ \epsilon$$
What would be solution for the more general case:
$$y=\beta X+ \...
- 155
0
votes
0
answers
31
views
high feature correlation but good OLS prediction
I have regression results where unconstrained OLS is near optimal - out of sample scores are almost the best when compared to some other constrained regression models. Although the ratio of number of ...
- 1
1
vote
1
answer
32
views
Can L1 Lasso regularization can produce L2 Ridge regularization results?
Today I learned that the L1 Lasso's lambda parameter can be adjusted from small to large, with small being that Lasso basically produces the same result as least squares (no regularization/ leaves ...
- 513
1
vote
2
answers
81
views
How to preprocess my stepwise regression using lasso/ridge?
I am struggling in the preprocessing of some analyses. I have a dataframe with around 100 observations and quite a few possible predictors (categorial and numerical data, about 20 in total). I am ...
- 13
0
votes
0
answers
23
views
What is $\hat\beta^{Lasso}$ in matrix form [duplicate]
We know that $\hat\beta^{ridge}= (X^TX+\lambda.I)^{-1}X^TY$ but I was wondering if there was a similar equation for $\hat\beta^{Lasso}$.
- 25
1
vote
0
answers
80
views
How to mathematically add ridge penalty to a loss function?
For this loss function:
$$L(w_1,w_2) = -6(0.1\cos(w_1) - w_1^2 - 0.5w_2^7 + \sin(w_2))e^{-w_1^2-w_2^2}$$
Assuming the X (features) and Y (target feature) are embedded within the coefficients, so you ...
- 413
4
votes
3
answers
344
views
When will L2 norm and squared L2 norm be equivalent in ridge regression?
I am doing ridge regression and am wondering why it uses squared L2 norm. This post Why l2 norm squared but l1 norm not squared? gave some thought about why using it. But a further question arises: ...
- 85
2
votes
1
answer
51
views
Notation for predicting $\hatβ$ in ridge regression
I have been reading around ridge regression and have come across two forms of $\hatβ$ in textbooks. Am I correct in believing that $(X^TX+\lambda I)^{-1} X^TY$ is the same as $RSS + \sum_{j=1}^{p} \...
- 25
4
votes
2
answers
174
views
Why there is high variance of gradients estimated in the short directions in regression?
I was trying to understand Ridge Regression and came across the following excerpt from Hastie et al. in The Elements of Statistical Learning (section 3.4.1, Page 67):
If we consider fitting a linear ...
- 43
0
votes
0
answers
48
views
Two ways to get rid of multicollinearity
I have a couple of questions concerning multicollinearity in a linear regression model $Y=X \beta + \epsilon$.
If the design matrix presents some multicollinearity i.e. $\det(X^TX) \approx 0$, we can ...
- 189
0
votes
0
answers
27
views
GCV for Ridge Regression with correlated data
I just want to double-check that my setup for using GCV to find the optimal penalty parameter $\lambda$ is correct.
The general regression model with n observations and k explanators, the first of ...
- 469
0
votes
1
answer
140
views
Ridge regression, large lambda results in smaller RMSE of the training data
I am training the ridge regression on a one-day sensor data using the closed-form solution where
$$ \beta=(X^TX+\lambda*I)^{-1}X^TY $$ and Matlab.
The $X$ is 15 polynomial time matrix. I created a ...
3
votes
1
answer
49
views
Coefficient in lasso regression objective and not in ridge regression
As we know, Lasso regression has an objective of the form:
$\min_w \frac{||Xw - y||_2^2}{2n} + \alpha||w||_1,$
and Ridge regression has the form:
$\min_w ||Xw - y||_2^2 + \alpha||w||_2^2.$
My question ...
- 33
1
vote
0
answers
41
views
Why does Ridge Regression affect certain coefficients differently? [duplicate]
From my understanding, Ridge regression tends to shrink coefficients towards 0 as lambda increases. However, it seems this is not always the case - for features which are more statistically ...
3
votes
0
answers
57
views
Ridge Regression Implementation [duplicate]
I'm working through the math of ridge regression and am having some trouble replicating some R functions. My question is NOT about R, per se, I will just use it here for a reproducible example but ...
- 193
3
votes
2
answers
175
views
Ridge Regression - what is the relationship between $\lambda$ and the data's noise?
Currently reviewing some problems in my ML class, and I came across this problem:
You estimate a ridge regression model with some data taken from your robot, and
find (using cross validation) and ...
- 31
12
votes
2
answers
836
views
Why l2 norm squared but l1 norm not squared?
In the Lasso, and ElasticNet, we use, as penalty, the l1 norm without squaring. But in the ElasticNet and Ridge, we use the l2 norm squared. Why is that, is there a particular reason (computational, ...
1
vote
0
answers
17
views
How to choose a predictive model between MSE and graphical plot (observed against predicted value)?
After regularisation with Lasso and Ridge, I currently have a model under each, taking MSE values as shown below,
MSE, Ridge = 0.1923102
MSE, Lasso = 0.1292252
Both models have the same number of ...
- 11
0
votes
0
answers
44
views
When should you use L1 vs. L2 regularization? [duplicate]
Can't seem to find a good explanation online of concrete examples of where you would use one over the other?
I also read somewhere that L1 is supposedly slower than L2, but not sure how that is since ...
- 513
1
vote
1
answer
39
views
How does ridge regression solve the multidimensionality problem if it doesn't assign zero to some coefficients
I want to understand how does ridge regression solve the multidimensionality problem (when number of X variable is higher than the number of observations)? It shrinks the coefficients by introducing ...
- 33
3
votes
0
answers
95
views
Equivalence between Gaussian Process Regression and Kernel Ridge Regression
Consider the model
$$ y(\mathbf{X}) = f(\mathbf{X}) + \epsilon, $$
where $\mathbf{X}$ is a given $n\times D$ matrix, and where $\epsilon \sim \mathcal{N}(0, \sigma^{2}I_{N})$ is iid Gaussian and is ...
- 311
1
vote
0
answers
39
views
practical way to run statistical test on the coefficients obtained from ridge or lasso
In the OLS, we can run t-tests on the coefficients obtained from linear regression, but how can we test on the coefficients we obtained in the case of lasso or ridge? What's the common practice in ...
- 163
1
vote
0
answers
34
views
Double-layered optimization to find optimal regularization parameter lambda for Ridge/LASSO
I have an overdetermined system of equations problem where n >> m and the OLS almost always finds an approximation instead of an exact solution. I already ...
- 915
1
vote
0
answers
45
views
Multicollinearity Market mix modeling
I want to know what can be the best approach to handle multicollinearity. I am building a regression model with just 4 independent but all important variables and am not able to control the VIF.
...
- 21
1
vote
0
answers
43
views
How do the ridge or lasso coefficient changes when we add more variables
Suppose we run ridge or lasso regression over a bunch of features. And now suppose we add one more feature into the regressions. What will happen to the coefficients of the "old" features?
I ...
- 357
1
vote
0
answers
38
views
Is it possible to create a 0 intercept ridge regression model?
I am working on implementing ridge regression for market mix modeling where I wish to use my own create base(UCM) instead of intercept, I had been using linear regression for this purpose but now my ...
- 21
0
votes
0
answers
42
views
Ridge and Lasso regression coefficient change when we change the scales of the variables
I am interested in the following question: suppose we run a ridge of a lasso model on a bunch of variables. Now if we multiple one of the variables $x_1$ by 2, what happens to the coefficients.
Some ...
- 357
2
votes
1
answer
153
views
Centering vs standardizing in ridge regression
I have read that to apply ridge regression, we first need to standardize the predictive variables. That is because the variables should be in a homogenous scale so that lambda has an effect of the ...
- 121
2
votes
2
answers
172
views
How does ridge regression reduce the variance of the estimates of $\beta$
In the scikit-learn library, Ridge class, there is a note that reads: "Regularization improves the conditioning of the problem and reduces the variance of the estimates."
Given the ...
- 121
1
vote
0
answers
27
views
Is standarization necessary for ridge regression?
Is variable normalization necessary in Ridge regression (for both X and y)? If so, what happens (mathematically) if we don't do it?
- 121
2
votes
0
answers
65
views
Intercept and slope of ridge regression model
When we compute a Ridge regression model, do we need to compute the intercept separately from the slopes? As you know, the estimated $\beta$ for the ridge regression model is given by:
$\hat \beta = (...
- 121
0
votes
1
answer
47
views
Linear Model with fixed Weights and Terms / Ridge Regression / Regularization in R
I am working on setting up regression models for prediction in psychometrics and ran into challenges with cross validation. Essentially, I would like to have cross validated linear regression models ...
- 1
3
votes
3
answers
90
views
Is the magnitude coefficient vector in Ridge regression monotonic in lambda?
recently an interesting question came up and while I would have intuitively said it is not, other students have now made a compelling case (while not being sure themselves).
For ridge (or l2 ...
- 270
1
vote
2
answers
232
views
Geometrical interpretation of why can't ridge regression shrink coefficients to 0?
To explain the difference between Ridge and Lasso regression, following diagram is used as it is claimed that Ridge regression cannot shrink the regression coefficients to 0:
But my question is, if ...
4
votes
2
answers
732
views
Why in the Ridge regression, the coefficients cannot be 0?
In the second answer (https://stats.stackexchange.com/a/368426/287815) to the question (Why will ridge regression not shrink some coefficients to zero like lasso?), the OP found out that,
$β = 𝑥𝑦/(𝑥...
2
votes
1
answer
99
views
What does it say about the data if ridge regression is not reducing multicollinearity?
I am predicting the salary to be offered to a new candidate for which I am concentrating on just continuous (9 in number) variables. Variables are as attached. When I ran OLS the coefficient for total ...
- 23
2
votes
0
answers
91
views
Efficient computation of ridge/lasso regression
n the book "Introduction to Statistical learning with R" https://hastie.su.domains/ISLR2/ISLRv2_website.pdf ,
authors say on page 247 -
"There are very efficient algorithms for fitting ...
- 41
0
votes
0
answers
87
views
Summation signs in maximum likelihood (ridge regression example)
I can't find how to deal with summation signs ($\sum$) when performing maximum likelihood estimation. I've encountered it in ridge regression:
$\frac{1}{n} \sum_{i=1}^n (y_i-\theta^Tx_i)^2 + \lambda\...
- 229
1
vote
0
answers
106
views
Force selected coefficients to be non-negative in ridge regression
I want to fit a ridge regression on ~ 47 variables and 12 of them I want to be positive (or at least non-negative).
I'm using sklearn and doing the following:
...
- 113
0
votes
1
answer
59
views
Effective degrees of freedom concrete example of variable distribution with Ridge
I am reading a report where they used Lasso and penalty term $\lambda$. Below is a table presented:
I have a question about the DF. The original model had 16 variables and no intercept, i.e without ...
- 229