Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
0
votes
0answers
6 views
Using Quadratic Programming to solve Lasso and Ridge regression models?
I'm trying to build linear, ridge and lasso regression models for at set of data (40 obs., 4 features, 1 response).
I'm building the models using the sklearn package for Python and I can easily find ...
0
votes
1answer
11 views
Interpreting ridge coefficients as a function of regularization
Data consists of 40 observations with 4 dimensions and a response-variable.
When doing a ridge regression on my data and plotting the coefficients and coefficient errors (MSE of the ridge ...
0
votes
3answers
44 views
Necessity of standardizing data in regularized regression [duplicate]
It is well known that in Ridge or LASSO regression we add a regularization term to penalize large regression coefficients. What if the true relationship between the response and covariates relies on a ...
0
votes
1answer
21 views
Initialization for ridge regression
What should be a better initialization for weights of ridge regression if I have to perform gradient descent. I have tried with all weights 0, all weights 1, and random initialization. In all the ...
3
votes
1answer
245 views
Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?
By my understanding, for a matrix with n samples and p features:
Ridge regression using Cholesky decomposition takes O(p^3) time
Ridge regression using SVD takes O(p^3) time
Computing SVD when only ...
2
votes
0answers
26 views
Why is it much quicker to compute ridge regression than regular linear regression?
By my understanding, for a matrix with n samples and p features:
Ridge regression using cholesky takes O(p^3) time
Ordinary linear regression takes O(p^3) time
Singular value decomposition if u, v ...
1
vote
0answers
17 views
lambda value threshold beyond which shrinkage penalty in glm is high and coefficients approach zero
Conceptually, for both LASSO and ridge regression methods as lambda becomes "very large", the penalty impact grows and the coefficients approach zero. However,
a) is there a particular threshold for ...
-1
votes
1answer
47 views
True or false? (Ridge regression has higher error rate than standard linear regression for test set)
When using ridge regression, we would expect the error/loss function on the test set to be higher than if we used standard linear regression with no penalty.
I know that for the training set, the ...
0
votes
0answers
6 views
choice of the lambda parameter in the logit multinomial ridge model
I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
1
vote
0answers
29 views
Is group lasso equivalent to ridge regression when there is 1 group
On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso).
However in group lasso we have norm ...
1
vote
1answer
29 views
What minimization problem has this solution
Consider the following basic minimization problem
\begin{equation}
{\displaystyle \min _{\beta\in R^{n}}{\frac {1}{n}}\|Y-X\beta\|_{R^{n}}^{2}},\end{equation}
with solution
\begin{equation}
{\beta^*=(...
0
votes
1answer
34 views
Why is L2 regression good for handling multicollinearity? [duplicate]
Looking for an intuitive explanation, thanks.
1
vote
1answer
26 views
glmnet package: “mgaussian” vs “gaussian” for $\alpha = 0$
In multiresponse Gaussian family the objective function when $\alpha = 0$:
\begin{align}
\frac{1}{2n}||Y-XB||_F^2 + \frac{\lambda}{2}||B||_F^2.
\end{align}
This can also mathematically solved as
\...
0
votes
0answers
18 views
Ridge, Lasso or Elastic nets used in Accounting Research
I am trying to come up with ideas for my master's thesis and was wondering why literature on the above mentioned regression methods within Accounting Research is non-existent?
I felt like the ...
0
votes
0answers
13 views
Penalized Regression: “ridge” RMSE and coefficients larger than those for plain “lm” [duplicate]
Working with the "prostate" dataset in "ElemStatLearn" package.
...
3
votes
0answers
62 views
How to convert Ridge regression into a constraint optimization problem? [duplicate]
I hope this question wasn't asked before, I was looking for an answer and didn't find one.
I'm trying to understand the Ridge regression problem.
If I understand correctly, Ridge regression is trying ...
1
vote
1answer
46 views
Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?
Say we have a dataset, X, which is Nx2 where N is the number of examples and 2 is the number of dimensions "features". If we were to run a kernel ridge regression (or SVM or whatever) on these ...
2
votes
0answers
52 views
Does Ridge regression always yield lower MSE value compared to OLS?
First time asking question in StackExchange after being a long time lurker.
I am trying to analyze some simple data using R. I found the best lambda for Ridge regression using cross-validation, then ...
1
vote
1answer
54 views
Is there any two-stage procedure for elastic net as LASSO?
I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables?
. It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased ...
0
votes
1answer
25 views
Kernelize Linear Regression
We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf.
However would it be possible to find a vector $\boldsymbol\alpha$...
0
votes
0answers
23 views
Selection criteria for penalty parameters in the ridge multinomial logit model
I appeal to you for the following doubt. I am adjusting a ridge multinomial logit model but I have problems in the criterion when choosing the lambda parameter that gives better results, besides ...
0
votes
1answer
101 views
R: What does train() do when it calculates ridge regression?
I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression.
I will write up the two methods and then pose my question
Initialize with the ...
0
votes
0answers
29 views
How is the generalization of LASSO called?
I know that ridge regression is a special case of Tykhonv regularization.
In fact with Tykhonov one tries to minimize:
$|| Ax - b ||^2 +|| \Gamma x ||^2$
If $\Gamma$ is the identity matrix scaled by ...
2
votes
0answers
62 views
Why not use Ridge after Lasso vs relaxed Lasso
Has anyone ever applied a ridge regression on a
model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
14 views
Multinomial logit with ridge penalization and value of time
I am fitting a multinomial logit model with ridge penalty and in turn estimating the value of time (VOT) or availability to pay (WTP). I want to work with real and simulated data.
For the real data I ...
0
votes
1answer
29 views
Logistic regression with coefficients penalized to other numbers
When you penalize logistic regression using l1, l2 or both penalizations, the coefficients are penalized towards 0. I would like to do the same thing but penalizing the coefficients towards other ...
0
votes
0answers
41 views
If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?
If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression?
I am centering my kernel as follows:
$$K_c(\mathbf{t}...
0
votes
0answers
30 views
Trouble understanding L1 and L2 cost function [duplicate]
When reading the Sklearn User Guide, one might see the following statement about Logistic Regression
As an optimization problem, binary class L2 penalized logistic
regression minimizes the ...
0
votes
0answers
34 views
Does standard error affected by the coefficient?
I make a comparison on ridge regression and OLS using simulation.
As i set my correlation as 0.9, which is high, i expect the standard error of ridge regression to be low. However, it is not. ...
0
votes
0answers
32 views
Tikhonov Regularization in continuous probability density functions
Can anyone helps me understand Bishop's Training with Noise is Equivalent to Tikhonov Regularization?
In the paper, Bishop first defined the cost function (Equation 1) as:
where $x$ stands for ...
1
vote
0answers
48 views
R code for robust ridge regression
I am having trouble in searching for the MSE value in using robust ridge regression. The robust estimators that i used is LTS and MM. However, when both robust estimators were applied to ridge, the ...
0
votes
1answer
40 views
Why can't ridge regression set slope to zero like LASSO does
I know that LASSO penalizes certain coefficents to zero by taking absolut value. However, ridge makes penalty by taking square instead. I am wondering why this difference forbid ridge from setting ...
0
votes
0answers
19 views
Ridge analytically vs glmnet [duplicate]
With an outcome variable and two correlated regressors...
...
0
votes
1answer
44 views
Before running a ridge regression model, do I need to preform variable selection?
I am currently constructing a model that uses last year's departmental information to predict employee churn for the current year. I have 55 features and 318 departments in my data set. A good ...
1
vote
0answers
39 views
Proof of variance of ridge estimate with only one predictor!
Let's consider Ridge with only one predictor (extreme and simple case). I would like to proof that $V(B_r)=\sigma^2/(1+\lambda)$, so its variance it less than OLS variance, that is $V(B_{OLS})=\sigma^...
1
vote
0answers
130 views
How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?
In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas:
Both of these methods attempt to improve generalization ...
2
votes
1answer
414 views
Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]
I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1].
To better understand it I'm reading Google's ...
0
votes
0answers
195 views
How to interpret/choose alpha in ridge regression
I have questions on how to apply ridge regression on my data set, which has about 75 samples with 8 features (x's) and usually 3 targets (y's).
I tried the following feature engineering methods.
...
3
votes
1answer
63 views
Why is individual R-squared higher than overall R-squared?
I ran a ridge regression model on a set of data across 6 groups.
As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-...
1
vote
1answer
46 views
Ridge Regression problem with output
I am trying to implement ridge regression in R, but the results are wrong.
$\hat\beta^{Ridge} = argmin\sum_{i=1}^N(y-\beta_0 - \sum_{j=1}^{p} x_{i,j}\beta_j)^2 + \lambda\sum_{j=1}^{p}\beta_j^2$
...
1
vote
1answer
171 views
Mean squared error (MSE) prediction performance: Ridge vs Lasso?
It says that the ridge will outperform lasso in terms of prediction performance when the prediction metric is MSE, according to the answer to this post below:
If only prediction is of interest, why ...
2
votes
1answer
305 views
Ridge/Lasso regression negative Lambda
I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
24 views
Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]
Background:
For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
0
votes
1answer
33 views
Adding more samples to ordinary regression is equall to ridge regression [duplicate]
I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
16
votes
2answers
375 views
LASSO and ridge from the Bayesian perspective: what about the tuning parameter?
Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. I guess (as I do not know enough about Bayesian statistics) that for a fixed ...
1
vote
0answers
34 views
How can I use the coefficient and important variables obtained from elastic net modelling [closed]
I have a big question here. Although I search over internet and also in research papers but couldn't find an answer to it.
I ran elastic net over a dataset that had close to 300 variables and a ...
3
votes
1answer
178 views
Showing that ridge regression is a solution to the following optimization problem [duplicate]
$$\hat{\theta}=\arg\min_{\theta}\{ ||y-X\theta||_2^2+\lambda||\theta||_2^2\},$$ where $X$ is an $n\times p$ matrix.
We have if $y=X\theta+\varepsilon$ then $$\hat{\theta}^{\text{ridge}}=(X^TX+\...
5
votes
1answer
532 views
Why lasso for feature selection?
Suppose I have a high-dimensional dataset and want to perform feature selection. One way is to train a model capable of identifying the most important features in this dataset and use this to throw ...
-1
votes
1answer
178 views
cv.glmnet vs glm vs lm.ridge
I am currently trying to build a ridge regression model, and knows that the lm.ridge, glm and cv.glmnet functions can enable me to do so. However, I really do not know what are the differences between ...
1
vote
1answer
49 views
Are the assumptions of subset selection violated of all the data comes from a single person?
I’m interested in using machine learning techniques such as subset selection, lasso and ridge regression to predict which words an individual kid will get wrong. I have about 300 predictors and all my ...
