Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

0
votes
0answers
6 views

Using Quadratic Programming to solve Lasso and Ridge regression models?

I'm trying to build linear, ridge and lasso regression models for at set of data (40 obs., 4 features, 1 response). I'm building the models using the sklearn package for Python and I can easily find ...
0
votes
1answer
11 views

Interpreting ridge coefficients as a function of regularization

Data consists of 40 observations with 4 dimensions and a response-variable. When doing a ridge regression on my data and plotting the coefficients and coefficient errors (MSE of the ridge ...
0
votes
3answers
44 views

Necessity of standardizing data in regularized regression [duplicate]

It is well known that in Ridge or LASSO regression we add a regularization term to penalize large regression coefficients. What if the true relationship between the response and covariates relies on a ...
0
votes
1answer
21 views

Initialization for ridge regression

What should be a better initialization for weights of ridge regression if I have to perform gradient descent. I have tried with all weights 0, all weights 1, and random initialization. In all the ...
3
votes
1answer
245 views

Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?

By my understanding, for a matrix with n samples and p features: Ridge regression using Cholesky decomposition takes O(p^3) time Ridge regression using SVD takes O(p^3) time Computing SVD when only ...
2
votes
0answers
26 views

Why is it much quicker to compute ridge regression than regular linear regression?

By my understanding, for a matrix with n samples and p features: Ridge regression using cholesky takes O(p^3) time Ordinary linear regression takes O(p^3) time Singular value decomposition if u, v ...
1
vote
0answers
17 views

lambda value threshold beyond which shrinkage penalty in glm is high and coefficients approach zero

Conceptually, for both LASSO and ridge regression methods as lambda becomes "very large", the penalty impact grows and the coefficients approach zero. However, a) is there a particular threshold for ...
-1
votes
1answer
47 views

True or false? (Ridge regression has higher error rate than standard linear regression for test set)

When using ridge regression, we would expect the error/loss function on the test set to be higher than if we used standard linear regression with no penalty. I know that for the training set, the ...
0
votes
0answers
6 views

choice of the lambda parameter in the logit multinomial ridge model

I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
1
vote
0answers
29 views

Is group lasso equivalent to ridge regression when there is 1 group

On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso). However in group lasso we have norm ...
1
vote
1answer
29 views

What minimization problem has this solution

Consider the following basic minimization problem \begin{equation} {\displaystyle \min _{\beta\in R^{n}}{\frac {1}{n}}\|Y-X\beta\|_{R^{n}}^{2}},\end{equation} with solution \begin{equation} {\beta^*=(...
0
votes
1answer
34 views

Why is L2 regression good for handling multicollinearity? [duplicate]

Looking for an intuitive explanation, thanks.
1
vote
1answer
26 views

glmnet package: “mgaussian” vs “gaussian” for $\alpha = 0$

In multiresponse Gaussian family the objective function when $\alpha = 0$: \begin{align} \frac{1}{2n}||Y-XB||_F^2 + \frac{\lambda}{2}||B||_F^2. \end{align} This can also mathematically solved as \...
0
votes
0answers
18 views

Ridge, Lasso or Elastic nets used in Accounting Research

I am trying to come up with ideas for my master's thesis and was wondering why literature on the above mentioned regression methods within Accounting Research is non-existent? I felt like the ...
0
votes
0answers
13 views

Penalized Regression: “ridge” RMSE and coefficients larger than those for plain “lm” [duplicate]

Working with the "prostate" dataset in "ElemStatLearn" package. ...
3
votes
0answers
62 views

How to convert Ridge regression into a constraint optimization problem? [duplicate]

I hope this question wasn't asked before, I was looking for an answer and didn't find one. I'm trying to understand the Ridge regression problem. If I understand correctly, Ridge regression is trying ...
1
vote
1answer
46 views

Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?

Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
2
votes
0answers
52 views

Does Ridge regression always yield lower MSE value compared to OLS?

First time asking question in StackExchange after being a long time lurker. I am trying to analyze some simple data using R. I found the best lambda for Ridge regression using cross-validation, then ...
1
vote
1answer
54 views

Is there any two-stage procedure for elastic net as LASSO?

I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables? . It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased ...
0
votes
1answer
25 views

Kernelize Linear Regression

We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf. However would it be possible to find a vector $\boldsymbol\alpha$...
0
votes
0answers
23 views

Selection criteria for penalty parameters in the ridge multinomial logit model

I appeal to you for the following doubt. I am adjusting a ridge multinomial logit model but I have problems in the criterion when choosing the lambda parameter that gives better results, besides ...
0
votes
1answer
101 views

R: What does train() do when it calculates ridge regression?

I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression. I will write up the two methods and then pose my question Initialize with the ...
0
votes
0answers
29 views

How is the generalization of LASSO called?

I know that ridge regression is a special case of Tykhonv regularization. In fact with Tykhonov one tries to minimize: $|| Ax - b ||^2 +|| \Gamma x ||^2$ If $\Gamma$ is the identity matrix scaled by ...
2
votes
0answers
62 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
14 views

Multinomial logit with ridge penalization and value of time

I am fitting a multinomial logit model with ridge penalty and in turn estimating the value of time (VOT) or availability to pay (WTP). I want to work with real and simulated data. For the real data I ...
0
votes
1answer
29 views

Logistic regression with coefficients penalized to other numbers

When you penalize logistic regression using l1, l2 or both penalizations, the coefficients are penalized towards 0. I would like to do the same thing but penalizing the coefficients towards other ...
0
votes
0answers
41 views

If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?

If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression? I am centering my kernel as follows: $$K_c(\mathbf{t}...
0
votes
0answers
30 views

Trouble understanding L1 and L2 cost function [duplicate]

When reading the Sklearn User Guide, one might see the following statement about Logistic Regression As an optimization problem, binary class L2 penalized logistic regression minimizes the ...
0
votes
0answers
34 views

Does standard error affected by the coefficient?

I make a comparison on ridge regression and OLS using simulation. As i set my correlation as 0.9, which is high, i expect the standard error of ridge regression to be low. However, it is not. ...
0
votes
0answers
32 views

Tikhonov Regularization in continuous probability density functions

Can anyone helps me understand Bishop's Training with Noise is Equivalent to Tikhonov Regularization? In the paper, Bishop first defined the cost function (Equation 1) as: where $x$ stands for ...
1
vote
0answers
48 views

R code for robust ridge regression

I am having trouble in searching for the MSE value in using robust ridge regression. The robust estimators that i used is LTS and MM. However, when both robust estimators were applied to ridge, the ...
0
votes
1answer
40 views

Why can't ridge regression set slope to zero like LASSO does

I know that LASSO penalizes certain coefficents to zero by taking absolut value. However, ridge makes penalty by taking square instead. I am wondering why this difference forbid ridge from setting ...
0
votes
0answers
19 views

Ridge analytically vs glmnet [duplicate]

With an outcome variable and two correlated regressors... ...
0
votes
1answer
44 views

Before running a ridge regression model, do I need to preform variable selection?

I am currently constructing a model that uses last year's departmental information to predict employee churn for the current year. I have 55 features and 318 departments in my data set. A good ...
1
vote
0answers
39 views

Proof of variance of ridge estimate with only one predictor!

Let's consider Ridge with only one predictor (extreme and simple case). I would like to proof that $V(B_r)=\sigma^2/(1+\lambda)$, so its variance it less than OLS variance, that is $V(B_{OLS})=\sigma^...
1
vote
0answers
130 views

How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?

In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas: Both of these methods attempt to improve generalization ...
2
votes
1answer
414 views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
0
votes
0answers
195 views

How to interpret/choose alpha in ridge regression

I have questions on how to apply ridge regression on my data set, which has about 75 samples with 8 features (x's) and usually 3 targets (y's). I tried the following feature engineering methods. ...
3
votes
1answer
63 views

Why is individual R-squared higher than overall R-squared?

I ran a ridge regression model on a set of data across 6 groups. As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-...
1
vote
1answer
46 views

Ridge Regression problem with output

I am trying to implement ridge regression in R, but the results are wrong. $\hat\beta^{Ridge} = argmin\sum_{i=1}^N(y-\beta_0 - \sum_{j=1}^{p} x_{i,j}\beta_j)^2 + \lambda\sum_{j=1}^{p}\beta_j^2$ ...
1
vote
1answer
171 views

Mean squared error (MSE) prediction performance: Ridge vs Lasso?

It says that the ridge will outperform lasso in terms of prediction performance when the prediction metric is MSE, according to the answer to this post below: If only prediction is of interest, why ...
2
votes
1answer
305 views

Ridge/Lasso regression negative Lambda

I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]

Background: For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
0
votes
1answer
33 views

Adding more samples to ordinary regression is equall to ridge regression [duplicate]

I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
16
votes
2answers
375 views

LASSO and ridge from the Bayesian perspective: what about the tuning parameter?

Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. I guess (as I do not know enough about Bayesian statistics) that for a fixed ...
1
vote
0answers
34 views

How can I use the coefficient and important variables obtained from elastic net modelling [closed]

I have a big question here. Although I search over internet and also in research papers but couldn't find an answer to it. I ran elastic net over a dataset that had close to 300 variables and a ...
3
votes
1answer
178 views

Showing that ridge regression is a solution to the following optimization problem [duplicate]

$$\hat{\theta}=\arg\min_{\theta}\{ ||y-X\theta||_2^2+\lambda||\theta||_2^2\},$$ where $X$ is an $n\times p$ matrix. We have if $y=X\theta+\varepsilon$ then $$\hat{\theta}^{\text{ridge}}=(X^TX+\...
5
votes
1answer
532 views

Why lasso for feature selection?

Suppose I have a high-dimensional dataset and want to perform feature selection. One way is to train a model capable of identifying the most important features in this dataset and use this to throw ...
-1
votes
1answer
178 views

cv.glmnet vs glm vs lm.ridge

I am currently trying to build a ridge regression model, and knows that the lm.ridge, glm and cv.glmnet functions can enable me to do so. However, I really do not know what are the differences between ...
1
vote
1answer
49 views

Are the assumptions of subset selection violated of all the data comes from a single person?

I’m interested in using machine learning techniques such as subset selection, lasso and ridge regression to predict which words an individual kid will get wrong. I have about 300 predictors and all my ...