Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
0
votes
1answer
34 views
R: What does train() do when it calculates ridge regression?
I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression.
I will write up the two methods and then pose my question
Initialize with the ...
0
votes
0answers
19 views
How is the generalization of LASSO called?
I know that ridge regression is a special case of Tykhonv regularization.
In fact with Tykhonov one tries to minimize:
$|| Ax - b ||^2 +|| \Gamma x ||^2$
If $\Gamma$ is the identity matrix scaled by ...
2
votes
0answers
29 views
Why not use Ridge after Lasso vs relaxed Lasso
Has anyone ever applied a ridge regression on a
model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
10 views
Multinomial logit with ridge penalization and value of time
I am fitting a multinomial logit model with ridge penalty and in turn estimating the value of time (VOT) or availability to pay (WTP). I want to work with real and simulated data.
For the real data I ...
0
votes
1answer
24 views
Logistic regression with coefficients penalized to other numbers
When you penalize logistic regression using l1, l2 or both penalizations, the coefficients are penalized towards 0. I would like to do the same thing but penalizing the coefficients towards other ...
0
votes
0answers
23 views
If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?
If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression?
I am centering my kernel as follows:
$$K_c(\mathbf{t}...
0
votes
0answers
29 views
Trouble understanding L1 and L2 cost function [duplicate]
When reading the Sklearn User Guide, one might see the following statement about Logistic Regression
As an optimization problem, binary class L2 penalized logistic
regression minimizes the ...
0
votes
0answers
31 views
Does standard error affected by the coefficient?
I make a comparison on ridge regression and OLS using simulation.
As i set my correlation as 0.9, which is high, i expect the standard error of ridge regression to be low. However, it is not. ...
0
votes
0answers
13 views
Tikhonov Regularization in continuous probability density functions
Can anyone helps me understand Bishop's Training with Noise is Equivalent to Tikhonov Regularization?
In the paper, Bishop first defined the cost function (Equation 1) as:
where $x$ stands for ...
0
votes
0answers
25 views
R code for robust ridge regression
I am having trouble in searching for the MSE value in using robust ridge regression. The robust estimators that i used is LTS and MM. However, when both robust estimators were applied to ridge, the ...
0
votes
0answers
20 views
Variance Inflation Factor for Ridge Regression model
Is there such a thing as a metric that can determine if multicollinearity is violated in a ridge, elastic net, or lasso model? From programmatic terms, if I have a glmnet package model, is there a way ...
0
votes
1answer
32 views
Why can't ridge regression set slope to zero like LASSO does
I know that LASSO penalizes certain coefficents to zero by taking absolut value. However, ridge makes penalty by taking square instead. I am wondering why this difference forbid ridge from setting ...
0
votes
0answers
17 views
Ridge analytically vs glmnet [duplicate]
With an outcome variable and two correlated regressors...
...
0
votes
1answer
38 views
Before running a ridge regression model, do I need to preform variable selection?
I am currently constructing a model that uses last year's departmental information to predict employee churn for the current year. I have 55 features and 318 departments in my data set. A good ...
1
vote
0answers
18 views
Proof of variance of ridge estimate with only one predictor!
Let's consider Ridge with only one predictor (extreme and simple case). I would like to proof that $V(B_r)=\sigma^2/(1+\lambda)$, so its variance it less than OLS variance, that is $V(B_{OLS})=\sigma^...
1
vote
0answers
50 views
How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?
In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas:
Both of these methods attempt to improve generalization ...
2
votes
1answer
145 views
Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]
I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1].
To better understand it I'm reading Google's ...
0
votes
0answers
30 views
Results of cross validation don't consistent, what it mean
I have a data set which has about 100 samples. Each sample has 9 features ($x_1, ..., x_9$) and one targets ($y$). I tried ridge regression on this data set using sklearn. In the regression, the cost ...
0
votes
0answers
99 views
How to interpret/choose alpha in ridge regression
I have questions on how to apply ridge regression on my data set, which has about 75 samples with 8 features (x's) and usually 3 targets (y's).
I tried the following feature engineering methods.
...
0
votes
0answers
54 views
Estimation of a variable in a linear equation
I have a linear equation: $a*x_1-b*x_2+x_3=1.0$. $a$ and $b$ are known parameters: $a=b=10e+6$. We used an approach to get the estimates of $x_1$ and $x_2$: $x_1=1.001$ and $x_2=0.9998$. We want to ...
3
votes
1answer
58 views
Why is individual R-squared higher than overall R-squared?
I ran a ridge regression model on a set of data across 6 groups.
As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-...
0
votes
0answers
75 views
Large Matrix to run cv.glmnet for multinomial
I am working on a large matrix with number of samples N=40 and features, P=7130. I am trying to fit the cv.glmnet() for the ridge but i am getting error while doing ...
1
vote
1answer
40 views
Ridge Regression problem with output
I am trying to implement ridge regression in R, but the results are wrong.
$\hat\beta^{Ridge} = argmin\sum_{i=1}^N(y-\beta_0 - \sum_{j=1}^{p} x_{i,j}\beta_j)^2 + \lambda\sum_{j=1}^{p}\beta_j^2$
...
1
vote
1answer
94 views
Mean squared error (MSE) prediction performance: Ridge vs Lasso?
It says that the ridge will outperform lasso in terms of prediction performance when the prediction metric is MSE, according to the answer to this post below:
If only prediction is of interest, why ...
2
votes
1answer
171 views
Ridge/Lasso regression negative Lambda
I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
24 views
Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]
Background:
For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
0
votes
1answer
32 views
Adding more samples to ordinary regression is equall to ridge regression [duplicate]
I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
16
votes
2answers
311 views
LASSO and ridge from the Bayesian perspective: what about the tuning parameter?
Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. I guess (as I do not know enough about Bayesian statistics) that for a fixed ...
1
vote
0answers
32 views
How can I use the coefficient and important variables obtained from elastic net modelling [closed]
I have a big question here. Although I search over internet and also in research papers but couldn't find an answer to it.
I ran elastic net over a dataset that had close to 300 variables and a ...
3
votes
1answer
98 views
Showing that ridge regression is a solution to the following optimization problem [duplicate]
$$\hat{\theta}=\arg\min_{\theta}\{ ||y-X\theta||_2^2+\lambda||\theta||_2^2\},$$ where $X$ is an $n\times p$ matrix.
We have if $y=X\theta+\varepsilon$ then $$\hat{\theta}^{\text{ridge}}=(X^TX+\...
4
votes
1answer
128 views
Why lasso for feature selection?
Suppose I have a high-dimensional dataset and want to perform feature selection. One way is to train a model capable of identifying the most important features in this dataset and use this to throw ...
-1
votes
1answer
102 views
cv.glmnet vs glm vs lm.ridge
I am currently trying to build a ridge regression model, and knows that the lm.ridge, glm and cv.glmnet functions can enable me to do so. However, I really do not know what are the differences between ...
1
vote
1answer
47 views
Are the assumptions of subset selection violated of all the data comes from a single person?
I’m interested in using machine learning techniques such as subset selection, lasso and ridge regression to predict which words an individual kid will get wrong. I have about 300 predictors and all my ...
5
votes
1answer
240 views
Is there any special case where ridge regression can shrink coefficients to zero?
Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ?
It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge ...
4
votes
1answer
123 views
Ridge regression to minimize RMSE instead of MSE
Cross-posted from my identical question on math.stackexchange:
Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} ...
1
vote
0answers
209 views
How to decide whether to use Ridge Regression/LASSO/Elastic Net or Random Forest for Feature Selection?
My understanding is rudimentary and high level but it seems like Ridge Regression/LASSO/Elastic Net would be better when the data is linear and Random Forest is better when the data is nonlinear? Also ...
1
vote
1answer
235 views
ridge and lasso models in caret with lambda=0
As far as I know, if I run a lasso model and a ridge model on the same data, and if i keep lambda=0, I'm getting the OLS.
Then, how is it possible that I get different results?
...
1
vote
0answers
69 views
L1 vs L2 stability?
See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/
"The instability property of the method of least absolute deviations means that, for a ...
2
votes
1answer
70 views
Ridge/Lasso for correlated response
I want to try a penalised linear regression (ridge/lasso) as a comparison to standard OLS for its predictive ability. My response variable is a continuous measure of an eye parameter, so there is (...
1
vote
1answer
49 views
Why parameters become zero in Group Lasso
I've already studied about Ridge, Lasso and Group Lasso. Lasso can estimate essential parameters. That is, some parameters got zero.
By the way, in group lasso, why parameters corresponding to a ...
3
votes
0answers
68 views
A theoretical explanation why ridge is superior to lasso in non-sparse models
There are several posts about the comparison of lasso vs. ridge. However I didn't find an explanation to my question. My question is why ridge is generating lower prediction errors in cases where the ...
-1
votes
1answer
44 views
Train MSE becomes smaller then test mse when model becomes morgen complex
Iam doing a ridge and lasso regression and choose my lambdas via cross validation with K = 5 and K = 10.
I do this with 3 data sets because i want to analize if more variables yield to a better ...
-1
votes
1answer
69 views
I have some few question regarding OLS.
Can I interpret my my coefficient's p-values even I violated the error normality assumptions? I have a large sample size.
2
votes
1answer
52 views
Ridge Regression as Robust Optimization
We were told to assume in class that the below optimization formulations are equivalent-
$$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$
$$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
0
votes
1answer
84 views
Why lasso yield a higher mse then ridge?
I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction ...
4
votes
0answers
109 views
Prediction with OLS better then prediction with lasso or ridge
I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation.
After that i wanted to compare the ...
1
vote
0answers
67 views
Relationship between regularization parameter in Ridge/Lasso with budget constraint
The equation for lasso and ridge regression are given as follows in the ISLR textbook:
The dual form of the above equations are given in terms of budget as below:
I am wondering if there is a ...
1
vote
0answers
345 views
Glmnet: How to select Lambda and Alpha
I'd like to pick the optimal lambda and alpha using the Glmnet package. I'm open to all models (Ridge, Lasso, Elastic). I'm assuming some out of sample error/cross validation is the best model ...
2
votes
1answer
205 views
Why weights are not negative in Lasso regression?
I can understand that lasso could make some weights to zero and prevent over-fitting. But for all the figure I see about lasso regression, weight will stay at zero once it reach zero and increasing ...
0
votes
0answers
50 views
Why is train MSE bigger then test MSE?
I did a lasso and ridge regression.
In both cases my train mse is bigger then my test mse and i ask myself why ?
