Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
718
questions
1
vote
0
answers
21
views
Multicollinearity Market mix modeling
I want to know what can be the best approach to handle multicollinearity. I am building a regression model with just 4 independent but all important variables and am not able to control the VIF.
...
- 21
1
vote
0
answers
32
views
How do the ridge or lasso coefficient changes when we add more variables
Suppose we run ridge or lasso regression over a bunch of features. And now suppose we add one more feature into the regressions. What will happen to the coefficients of the "old" features?
I ...
- 213
1
vote
0
answers
22
views
Is it possible to create a 0 intercept ridge regression model?
I am working on implementing ridge regression for market mix modeling where I wish to use my own create base(UCM) instead of intercept, I had been using linear regression for this purpose but now my ...
- 21
0
votes
0
answers
13
views
Ridge and Lasso regression coefficient change when we change the scales of the variables
I am interested in the following question: suppose we run a ridge of a lasso model on a bunch of variables. Now if we multiple one of the variables $x_1$ by 2, what happens to the coefficients.
Some ...
- 213
2
votes
1
answer
27
views
Centering vs standardizing in ridge regression
I have read that to apply ridge regression, we first need to standardize the predictive variables. That is because the variables should be in a homogenous scale so that lambda has an effect of the ...
- 111
2
votes
2
answers
76
views
How does ridge regression reduce the variance of the estimates of $\beta$
In the scikit-learn library, Ridge class, there is a note that reads: "Regularization improves the conditioning of the problem and reduces the variance of the estimates."
Given the ...
- 111
1
vote
0
answers
25
views
Is standarization necessary for ridge regression?
Is variable normalization necessary in Ridge regression (for both X and y)? If so, what happens (mathematically) if we don't do it?
- 111
2
votes
0
answers
18
views
Intercept and slope of ridge regression model
When we compute a Ridge regression model, do we need to compute the intercept separately from the slopes? As you know, the estimated $\beta$ for the ridge regression model is given by:
$\hat \beta = (...
- 111
0
votes
0
answers
8
views
Linear Model with fixed Weights and Terms / Ridge Regression / Regularization in R
I am working on setting up regression models for prediction in psychometrics and ran into challenges with cross validation. Essentially, I would like to have cross validated linear regression models ...
- 1
3
votes
3
answers
46
views
Is the magnitude coefficient vector in Ridge regression monotonic in lambda?
recently an interesting question came up and while I would have intuitively said it is not, other students have now made a compelling case (while not being sure themselves).
For ridge (or l2 ...
- 229
0
votes
1
answer
63
views
Geometrical interpretation of why can't ridge regression shrink coefficients to 0?
To explain the difference between Ridge and Lasso regression, following diagram is used as it is claimed that Ridge regression cannot shrink the regression coefficients to 0:
But my question is, if ...
3
votes
2
answers
343
views
Why in the Ridge regression, the coefficients cannot be 0?
In the second answer (https://stats.stackexchange.com/a/368426/287815) to the question (Why will ridge regression not shrink some coefficients to zero like lasso?), the OP found out that,
$β = 𝑥𝑦/(𝑥...
2
votes
1
answer
74
views
What does it say about the data if ridge regression is not reducing multicollinearity?
I am predicting the salary to be offered to a new candidate for which I am concentrating on just continuous (9 in number) variables. Variables are as attached. When I ran OLS the coefficient for total ...
- 23
2
votes
0
answers
59
views
Efficient computation of ridge/lasso regression
n the book "Introduction to Statistical learning with R" https://hastie.su.domains/ISLR2/ISLRv2_website.pdf ,
authors say on page 247 -
"There are very efficient algorithms for fitting ...
- 41
0
votes
0
answers
24
views
Summation signs in maximum likelihood (ridge regression example)
I can't find how to deal with summation signs ($\sum$) when performing maximum likelihood estimation. I've encountered it in ridge regression:
$\frac{1}{n} \sum_{i=1}^n (y_i-\theta^Tx_i)^2 + \lambda\...
- 93
1
vote
0
answers
30
views
Force selected coefficients to be non-negative in ridge regression
I want to fit a ridge regression on ~ 47 variables and 12 of them I want to be positive (or at least non-negative).
I'm using sklearn and doing the following:
...
- 13
0
votes
1
answer
29
views
Effective degrees of freedom concrete example of variable distribution with Ridge
I am reading a report where they used Lasso and penalty term $\lambda$. Below is a table presented:
I have a question about the DF. The original model had 16 variables and no intercept, i.e without ...
- 93
0
votes
0
answers
17
views
Interpretation of lasso shrinkage
In the case of the ridge estimator, we can interpret the shrinkage induced by the ridge estimator to be at its most extreme when a predictor has low variance. High-variance predictors provide the most ...
- 187
4
votes
1
answer
33
views
Should interactions also be scaled in LASSO/Ridge, or just constituent covariates?
I understand that in LASSO/Ridge it is best practice to scale covariates so that no single covariate dominates the penalized norm. However, when entering interaction terms, it is unclear whether only ...
- 65
1
vote
2
answers
75
views
Ridge Regression/Lasso
I have a dataset where I am trying to identify what group of people (i.e the predictors) are most likely to do X. I have ~25 predictors, ~5k cases and a binary outcome Y.
The predictors are ...
- 13
1
vote
1
answer
82
views
Why doesn't $\lambda=1$ in ridge regression?
Take traditional Ridge regression,
$$ Y_i = \sum_{j=0}^m \beta_{j} X_{i,j} + \epsilon_i $$
we minimize
$$ L_{ridge} = \arg \min_\hat{\beta}(\lambda||\beta||_2^2 + ||\epsilon||^2)$$
where $\lambda$ is ...
- 171
0
votes
1
answer
38
views
Ridge regression derivation from Murphy Machine Learning
Ridge regression, used to prevent overfitting, penalizes the coefficients $w_i$ of linear regression if they are too large. It is the solution to the problem
$$\arg\max_\textbf w \sum_{i=1}^N \ln \...
- 533
1
vote
1
answer
54
views
Kernel trick implemented for Ridge Regression
I am trying to see the kernel trick implemented for Ridge Regression.
As a first step, I want to rewrite the solution of Ridge regression. I know that:
$ \hat{\beta} = (X^TX + \lambda I_n)^{-1} X^T Y $...
- 41
0
votes
0
answers
24
views
Ridge estimator for orthogonal basis
In the book "Elements of Statistical Learning", the author compares the OSL estimator with Lasso, Ridge and Best Subset for the special case of Orthogonal X. I am attaching the particular ...
0
votes
0
answers
45
views
Linear and Ridge regression yield the same R² and MSE
I'm currently practicing on the NY taxi dataset but I'm having an issue and I'm sure it's because of some stupid mistake.
After cleaning the dataset, I'm taking the following features and try to ...
- 25
1
vote
0
answers
38
views
Understanding an equation in Pattern Recognition and Machine Learning from Bishop [duplicate]
I just started with Machine Learning and the statistics behind it, thus I am trying to understand as much derivation as possible when I see some formulae or resulting variables.
Today I stumbled upon ...
- 73
5
votes
1
answer
113
views
Number of samples in scikit-Learn cost function for Ridge/Lasso regression
I am using scikit-learn to train some regression models on data and noticed that the cost function for Lasso Regression is defined like this:
,
whereas the cost function for e.g. Ridge Regression is ...
- 51
0
votes
0
answers
20
views
Detailed comparison of two methods for obtaining the ridge regression solution
I have come across two different ways of obtaining the ridge regression solution, which are as follows:
Method1:-(obtained from here)
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
- 187
3
votes
0
answers
103
views
Ridge Regression Alpha/Lambda: Basic Characteristics?
I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (...
- 205
6
votes
1
answer
248
views
Ridge or multiple linear regression following PCA?
I have a real world clinical dataset with a severe issue of p >> n. I have thus decided to run PCA before modelling the data. This leads to a dataset with 150 samples with 85 features.
I would ...
- 315
0
votes
0
answers
34
views
Ridge Regression: Ridge traces are barely change across different k values [EDIT]
I am working with Matlab and following the example laid out here:
https://www.mathworks.com/help/stats/ridge.html
I then use the code above on my own data (with ~ 80 features). However, no matter what ...
- 315
1
vote
1
answer
59
views
Ridge classification: Interpreting prediction
I'm particularly concerned about the following problem when using ridge classification for predicting binary outcome
When I'm encoding the binary outcome as 1 and 0; my model accuracy is 0.6456
When ...
- 41
1
vote
1
answer
185
views
Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge outputs correlations
Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge regression outputs are equal to the correlations used in correlation ...
- 31
1
vote
0
answers
80
views
How is Cholesky decomposition used in ridge regression?
As far as I learnt, Cholesky decomposition can be used only for symmetrical positive definite matrices, but I can see it is used as solver in Sklearn-Ridge package, can somebody explain how it is used ...
- 11
0
votes
3
answers
110
views
Why does test MSE always decrease with increasing training size (and decreasing test size)?
Context:
I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
0
votes
0
answers
64
views
Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?
From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
- 11
0
votes
1
answer
61
views
How to extract MSEP or RMSEP from lassoCV?
I'm doing lasso and ridge regression in R with the package chemometrics. With ridgeCV it is easy to extract the SEP and MSEP values by ...
- 21
1
vote
0
answers
119
views
Mixed model via ridge regression
A mixed model can be recast as a ridge regression for a specific regularization parameter $\lambda$ that penalizes only the random effects -- aka dummy variables for the grouping levels.
Fitting a ...
- 12.4k
1
vote
2
answers
53
views
Introduction to Statistical Learning Eq. 6.12 and 6.13
Can someone please explain me how the optimization of 6.12 leads to 6.14 and the optimization of 6.13 leads to 6.15?
- 93
0
votes
1
answer
27
views
Ridge regression coefficients show model importance but the model evaulation not
I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
- 354
1
vote
0
answers
63
views
What is the consequence of "copying" a dataset for Ridge or Lasso?
So I know that for OLS, "copying" each of the N observations $(X_i,Y_i)$ once to get a dataset of size 2N has no effect on the values of the coefficients in OLS (related question).
Does this ...
- 161
6
votes
2
answers
748
views
Lasso coefficient for some features is higher than Linear Regression Coefficient
I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, ...
0
votes
1
answer
496
views
Is Ridge more robust than Lasso on feature selection?
My goal is to identify the best n-feature linear model, i.e. pick the model with only n-feature from total N features (n < N) and lowest Mean-Squared-Error (MSE). The experiment is on the Lasso and ...
- 63
0
votes
0
answers
151
views
Normalization and RidgeCV in Sklearn Pipeline - possible data leakage?
To avoid data leakage between the train and test set, I'm using sklearn's Pipeline as follows:
...
- 1
2
votes
1
answer
161
views
What would be the exact function for a ridge logistic regression with multiple variables?
I am looking for the correct equation for a ridge logistic regression for multiple variables. I thought it simply was:
$$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}$$
with an ...
- 354
0
votes
1
answer
93
views
How to "choose" binary variables which have a big impact on a regression?
I am currently facing an issue with analyzing my data for a project.
I have a dataset of about 100.000 samples. I have approximate 50 columns which are all binary and my dependent variable is time ...
0
votes
0
answers
30
views
In ridge regression, Why choose regression vector which has a minimum length?
As I reading a thesis named 'Ridge Regression: Biased Estimation for Nonorthogonal Problem' written by Hoerl and Kennard, I was struck by the below problem.
Let $\boldsymbol{B}$ be any estimate of the ...
- 21
1
vote
0
answers
150
views
Standard Error of Ridge regression
Why is it advised to use bootstrap for finding the SE for the ridge regression estimator?
Using the above formula, we can get the SE much faster than bootstrap. But all the research papers in ...
- 11
5
votes
1
answer
216
views
Ridge regression subtlety on intercept
I just noticed that when using ridge regression, there is a small subtlety on the penalised parameters, namely, we don't penalise $\theta_0$. Can someone give me a simple and intuitive explanation of ...
- 113
3
votes
3
answers
446
views
Why RidgeClassifier can be significantly faster than LogisticRegression with a high number of classes?
In Scikit document, we can find this statement
The RidgeClassifier can be significantly faster than e.g. LogisticRegression with a high number of classes because it can compute the projection matrix $...
- 257