Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
478 questions
3
votes
0answers
33 views
Why does shrinkage really work, what's so special about 0?
There is already a post on this site talking about the same issue:
Why does shrinkage work?
But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
3
votes
1answer
86 views
+100
Rademacher Bound, An Alternative to Cross Validation for Ridge?
Below is a theorem from the book "Foundations of Machine Learning".
It specifies the generalization bounds for Kernel Ridge Regression by making use of the Rademacher Complexity on linear models. $R(...
2
votes
1answer
35 views
Alternatives to Pre-Scaling Predictors in Lasso/Ridge Regression?
In lasso/ridge regression it's often recommended to scale predictors $X$ before estimation so that the coefficient estimates $\hat{\beta}$ will be invariant to the scale of predictors $X$.
Q: Is ...
0
votes
0answers
25 views
R - Gamma estimates in Kernel Ridge Regression
I am running a Kernel Ridge Regression in R. Mathematically, the minimization problem to be solved is the following:
$$ \min_{\boldsymbol{\beta} \in \mathbb{R}^{d}} \ \sum_{i = 1}^{n} (y_{i} - \left \...
0
votes
0answers
12 views
Ridge regression minimizing validation error
I have a question to a common deep learning problem, but I'm quite new to that field. Given a dataset of enough datas to represent a model, but we don't know whether the data is statistically ...
5
votes
2answers
215 views
One-to-one correspondence between penalty parameters of equivalent formulations of penalised regression methods
Ridge, LASSO and Elastic Net are three very popular methods of penalised regressions. All of these have more than one formulations. For example, two formulations for Ridge are:
minimise $\lVert Y - X ...
0
votes
1answer
36 views
Performing ridge regression using `optim`. Not sure where this extra term comes from?
Doing problems from ISLR and I've taken up the task of trying to do linear regression (and by extension lasso and ridge regression) using R's ...
3
votes
0answers
91 views
Nonnegative identity-link Poisson regression with ridge or fused ridge penalty
I would like to fit nonnegative identity-link Poisson regression models with a ridge or fused ridge penalty, i.e. with nonnegativity constraints on the fitted coefficients, Poisson error noise & a ...
1
vote
1answer
25 views
What is the constrain in ridge regression?
My class started learning about ridge regression two weeks ago. Before that we learned about Lagrange multipliers and the connection between that and ridge penalty/constrain function.
Ridge:
...
1
vote
1answer
40 views
Showing estimator is biased without assuming $X^TX$ is invertible?
I would like to show that the ridge regression estimator:
$$\beta^R = (X^TX+\lambda I)^{-1}X^T Y$$
is biased, where $Y \sim N(X\beta, \sigma^2 I)$.
If we assume that $X^TX$ is invertible, this can ...
3
votes
1answer
55 views
What is the relationship between the sum of squares of all weights and lambda in the ridge regression [duplicate]
Currently I am reading chapter 8, regression.
And I feel quite confused about the following paragraph(see picture below), does it mean in ridge algorithm, the sum of all weights will be less than ...
0
votes
0answers
26 views
Regularization techniques (ridge, lasso or elastic-net regression) for meta-analysis
I was perusing the StatQuest YouTube channel, and serendipitously listened to a series of videos on regularization techniques (ridge, lasso or elastic-net regression).
It came to my mind that such ...
0
votes
0answers
15 views
scaling maximum likelihood function with the amount of observations
I am a bit confused about the formulation of the maximum liklihood equation for logistic regression for ridge regression (and similar for lasso regression).
Where andrew Ng (coursera course) states ...
1
vote
2answers
73 views
How to interpret / metric Lasso regression coefficients
Edited Question, since it was a duplicate
I used Matlab to make a lasso model for my data that has 41 predictors and 1 response variable, and perhaps I used more variables that I need too or maybe ...
3
votes
0answers
23 views
Can one use ridge regression to test hypotheses? [duplicate]
Are the parameters (the beta's) that result from a ridge regression interpretable as we normally do with canonical linear regression? Can I use them to test hypothesis? Or is ridge regression ...
1
vote
1answer
68 views
why ridge regression only decreases slope and not increases it?
I was following the below example from 'StatQuest with Josh Starmer' youtube channel.
The example is pretty simple: red line is the usual 'least squares' (for the red points), and the blue one is ...
10
votes
2answers
443 views
Showing the Equivalence Between the $ {L}_{2} $ Norm Regularized Regression and $ {L}_{2} $ Norm Constrained Regression Using KKT
According to the references Book 1, Book 2 and paper.
It has been mentioned that there is an equivalence between the regularized regression (Ridge, LASSO and Elastic Net) and their constraint ...
2
votes
1answer
76 views
How to compute the change of Ridge regression solution when one row of data changes?
I understand that $\boldsymbol{\beta} = (X^TX + \lambda I)^{-1}X^T\mathbf{
Y}$ is the closed form solution of Ridge regression.
So sometimes, when I run a rolling window, meaning everytime I run the ...
0
votes
2answers
64 views
Feature selection using PCA for linear regression
I am using PCA to the training data set to do feature selection before applying linear regression to build a classifier model.
In this scenario, would it be useful to use ridge regression to ensure ...
0
votes
1answer
14 views
the constant coefficients are penalized in the ridge logit conditional model?
I am estimating a conditional ridge logit model, there is very little bibliography about it, and I do not know if the constant coefficients are penalized. My model has 2 variables and 3 alternatives, ...
2
votes
1answer
212 views
Ridge Regression with Gradient Descent Converges to OLS estimates
I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression.
This is ...
0
votes
0answers
29 views
Using Quadratic Programming to solve Lasso and Ridge regression models?
I'm trying to build linear, ridge and lasso regression models for at set of data (40 obs., 4 features, 1 response).
I'm building the models using the sklearn package for Python and I can easily find ...
0
votes
1answer
25 views
Interpreting ridge coefficients as a function of regularization
Data consists of 40 observations with 4 dimensions and a response-variable.
When doing a ridge regression on my data and plotting the coefficients and coefficient errors (MSE of the ridge ...
0
votes
3answers
47 views
Necessity of standardizing data in regularized regression [duplicate]
It is well known that in Ridge or LASSO regression we add a regularization term to penalize large regression coefficients. What if the true relationship between the response and covariates relies on a ...
0
votes
1answer
28 views
Initialization for ridge regression
What should be a better initialization for weights of ridge regression if I have to perform gradient descent. I have tried with all weights 0, all weights 1, and random initialization. In all the ...
3
votes
1answer
302 views
Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?
By my understanding, for a matrix with n samples and p features:
Ridge regression using Cholesky decomposition takes O(p^3) time
Ridge regression using SVD takes O(p^3) time
Computing SVD when only ...
3
votes
0answers
43 views
Why is it much quicker to compute ridge regression than regular linear regression?
By my understanding, for a matrix with n samples and p features:
Ridge regression using cholesky takes O(p^3) time
Ordinary linear regression takes O(p^3) time
Singular value decomposition if u, v ...
1
vote
0answers
26 views
lambda value threshold beyond which shrinkage penalty in glm is high and coefficients approach zero
Conceptually, for both LASSO and ridge regression methods as lambda becomes "very large", the penalty impact grows and the coefficients approach zero. However,
a) is there a particular threshold for ...
-1
votes
1answer
75 views
True or false? (Ridge regression has higher error rate than standard linear regression for test set)
When using ridge regression, we would expect the error/loss function on the test set to be higher than if we used standard linear regression with no penalty.
I know that for the training set, the ...
0
votes
0answers
7 views
choice of the lambda parameter in the logit multinomial ridge model
I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
1
vote
0answers
35 views
Is group lasso equivalent to ridge regression when there is 1 group
On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso).
However in group lasso we have norm ...
1
vote
1answer
30 views
What minimization problem has this solution
Consider the following basic minimization problem
\begin{equation}
{\displaystyle \min _{\beta\in R^{n}}{\frac {1}{n}}\|Y-X\beta\|_{R^{n}}^{2}},\end{equation}
with solution
\begin{equation}
{\beta^*=(...
0
votes
1answer
43 views
Why is L2 regression good for handling multicollinearity? [duplicate]
Looking for an intuitive explanation, thanks.
1
vote
1answer
47 views
glmnet package: “mgaussian” vs “gaussian” for $\alpha = 0$
In multiresponse Gaussian family the objective function when $\alpha = 0$:
\begin{align}
\frac{1}{2n}||Y-XB||_F^2 + \frac{\lambda}{2}||B||_F^2.
\end{align}
This can also mathematically solved as
\...
1
vote
0answers
22 views
Ridge, Lasso or Elastic nets used in Accounting Research
I am trying to come up with ideas for my master's thesis and was wondering why literature on the above mentioned regression methods within Accounting Research is non-existent?
I felt like the ...
0
votes
0answers
15 views
Penalized Regression: “ridge” RMSE and coefficients larger than those for plain “lm” [duplicate]
Working with the "prostate" dataset in "ElemStatLearn" package.
...
3
votes
0answers
63 views
How to convert Ridge regression into a constraint optimization problem? [duplicate]
I hope this question wasn't asked before, I was looking for an answer and didn't find one.
I'm trying to understand the Ridge regression problem.
If I understand correctly, Ridge regression is trying ...
1
vote
1answer
139 views
Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?
Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
2
votes
0answers
61 views
Does Ridge regression always yield lower MSE value compared to OLS?
First time asking question in StackExchange after being a long time lurker.
I am trying to analyze some simple data using R. I found the best lambda for Ridge regression using cross-validation, then ...
1
vote
1answer
80 views
Is there any two-stage procedure for elastic net as LASSO?
I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables?
. It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased ...
0
votes
1answer
28 views
Kernelize Linear Regression
We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf.
However would it be possible to find a vector $\boldsymbol\alpha$...
0
votes
0answers
29 views
Selection criteria for penalty parameters in the ridge multinomial logit model
I appeal to you for the following doubt. I am adjusting a ridge multinomial logit model but I have problems in the criterion when choosing the lambda parameter that gives better results, besides ...
0
votes
1answer
197 views
R: What does train() do when it calculates ridge regression?
I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression.
I will write up the two methods and then pose my question
Initialize with the ...
0
votes
0answers
37 views
How is the generalization of LASSO called?
I know that ridge regression is a special case of Tykhonv regularization.
In fact with Tykhonov one tries to minimize:
$|| Ax - b ||^2 +|| \Gamma x ||^2$
If $\Gamma$ is the identity matrix scaled by ...
2
votes
0answers
115 views
Why not use Ridge after Lasso vs relaxed Lasso
Has anyone ever applied a ridge regression on a
model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
15 views
Multinomial logit with ridge penalization and value of time
I am fitting a multinomial logit model with ridge penalty and in turn estimating the value of time (VOT) or availability to pay (WTP). I want to work with real and simulated data.
For the real data I ...
0
votes
1answer
37 views
Logistic regression with coefficients penalized to other numbers
When you penalize logistic regression using l1, l2 or both penalizations, the coefficients are penalized towards 0. I would like to do the same thing but penalizing the coefficients towards other ...
0
votes
0answers
51 views
If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?
If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression?
I am centering my kernel as follows:
$$K_c(\mathbf{t}...
0
votes
0answers
32 views
Trouble understanding L1 and L2 cost function [duplicate]
When reading the Sklearn User Guide, one might see the following statement about Logistic Regression
As an optimization problem, binary class L2 penalized logistic
regression minimizes the ...
0
votes
0answers
34 views
Does standard error affected by the coefficient?
I make a comparison on ridge regression and OLS using simulation.
As i set my correlation as 0.9, which is high, i expect the standard error of ridge regression to be low. However, it is not. ...
