Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A regularization method for regression models that shrinks coefficients towards zero.

1
vote
0answers
33 views

How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?

In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas: Both of these methods attempt to improve generalization ...
2
votes
1answer
33 views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
0
votes
0answers
30 views

Results of cross validation don't consistent, what it mean

I have a data set which has about 100 samples. Each sample has 9 features ($x_1, ..., x_9$) and one targets ($y$). I tried ridge regression on this data set using sklearn. In the regression, the cost ...
0
votes
0answers
50 views

How to interpret/choose alpha in ridge regression

I have questions on how to apply ridge regression on my data set, which has about 75 samples with 8 features (x's) and usually 3 targets (y's). I tried the following feature engineering methods. ...
0
votes
0answers
54 views

Estimation of a variable in a linear equation

I have a linear equation: $a*x_1-b*x_2+x_3=1.0$. $a$ and $b$ are known parameters: $a=b=10e+6$. We used an approach to get the estimates of $x_1$ and $x_2$: $x_1=1.001$ and $x_2=0.9998$. We want to ...
3
votes
1answer
55 views

Why is individual R-squared higher than overall R-squared?

I ran a ridge regression model on a set of data across 6 groups. As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-...
0
votes
0answers
33 views

Large Matrix to run cv.glmnet for multinomial

I am working on a large matrix with number of samples N=40 and features, P=7130. I am trying to fit the cv.glmnet() for the ridge but i am getting error while doing ...
1
vote
1answer
37 views

Ridge Regression problem with output

I am trying to implement ridge regression in R, but the results are wrong. $\hat\beta^{Ridge} = argmin\sum_{i=1}^N(y-\beta_0 - \sum_{j=1}^{p} x_{i,j}\beta_j)^2 + \lambda\sum_{j=1}^{p}\beta_j^2$ ...
1
vote
1answer
56 views

Mean squared error (MSE) prediction performance: Ridge vs Lasso?

It says that the ridge will outperform lasso in terms of prediction performance when the prediction metric is MSE, according to the answer to this post below: If only prediction is of interest, why ...
2
votes
1answer
92 views

Ridge/Lasso regression negative Lambda

I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
24 views

Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$? [duplicate]

Background: For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ ...
0
votes
1answer
31 views

Adding more samples to ordinary regression is equall to ridge regression [duplicate]

I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
6
votes
1answer
102 views

LASSO and ridge from the Bayesian perspective: what about the tuning parameter?

Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. I guess (as I do not know enough about Bayesian statistics) that for a fixed ...
1
vote
0answers
29 views

How can I use the coefficient and important variables obtained from elastic net modelling [closed]

I have a big question here. Although I search over internet and also in research papers but couldn't find an answer to it. I ran elastic net over a dataset that had close to 300 variables and a ...
3
votes
1answer
88 views

Showing that ridge regression is a solution to the following optimization problem [duplicate]

$$\hat{\theta}=\arg\min_{\theta}\{ ||y-X\theta||_2^2+\lambda||\theta||_2^2\},$$ where $X$ is an $n\times p$ matrix. We have if $y=X\theta+\varepsilon$ then $$\hat{\theta}^{\text{ridge}}=(X^TX+\...
4
votes
1answer
78 views

Why lasso for feature selection?

Suppose I have a high-dimensional dataset and want to perform feature selection. One way is to train a model capable of identifying the most important features in this dataset and use this to throw ...
-1
votes
1answer
65 views

cv.glmnet vs glm vs lm.ridge

I am currently trying to build a ridge regression model, and knows that the lm.ridge, glm and cv.glmnet functions can enable me to do so. However, I really do not know what are the differences between ...
1
vote
1answer
43 views

Are the assumptions of subset selection violated of all the data comes from a single person?

I’m interested in using machine learning techniques such as subset selection, lasso and ridge regression to predict which words an individual kid will get wrong. I have about 300 predictors and all my ...
5
votes
1answer
216 views

Is there any special case where ridge regression can shrink coefficients to zero?

Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ? It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge ...
4
votes
1answer
92 views

Ridge regression to minimize RMSE instead of MSE

Cross-posted from my identical question on math.stackexchange: Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} ...
1
vote
0answers
101 views

How to decide whether to use Ridge Regression/LASSO/Elastic Net or Random Forest for Feature Selection?

My understanding is rudimentary and high level but it seems like Ridge Regression/LASSO/Elastic Net would be better when the data is linear and Random Forest is better when the data is nonlinear? Also ...
1
vote
1answer
137 views

ridge and lasso models in caret with lambda=0

As far as I know, if I run a lasso model and a ridge model on the same data, and if i keep lambda=0, I'm getting the OLS. Then, how is it possible that I get different results? ...
1
vote
0answers
35 views

L1 vs L2 stability?

See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/ "The instability property of the method of least absolute deviations means that, for a ...
2
votes
1answer
54 views

Ridge/Lasso for correlated response

I want to try a penalised linear regression (ridge/lasso) as a comparison to standard OLS for its predictive ability. My response variable is a continuous measure of an eye parameter, so there is (...
-1
votes
0answers
16 views

Deriving the Ridge approximation for the lasso variance

I am trying to derive the ridge approximation formula for the lasso variance, but I am stucking at one point. As statet in the originial paper of tibshirani the penalty term can be transformed to $\...
1
vote
1answer
42 views

Why parameters become zero in Group Lasso

I've already studied about Ridge, Lasso and Group Lasso. Lasso can estimate essential parameters. That is, some parameters got zero. By the way, in group lasso, why parameters corresponding to a ...
2
votes
0answers
47 views

A theoretical explanation why ridge is superior to lasso in non-sparse models

There are several posts about the comparison of lasso vs. ridge. However I didn't find an explanation to my question. My question is why ridge is generating lower prediction errors in cases where the ...
-1
votes
1answer
29 views

Train MSE becomes smaller then test mse when model becomes morgen complex

Iam doing a ridge and lasso regression and choose my lambdas via cross validation with K = 5 and K = 10. I do this with 3 data sets because i want to analize if more variables yield to a better ...
-1
votes
1answer
69 views

I have some few question regarding OLS.

Can I interpret my my coefficient's p-values even I violated the error normality assumptions? I have a large sample size.
2
votes
1answer
45 views

Ridge Regression as Robust Optimization

We were told to assume in class that the below optimization formulations are equivalent- $$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$ $$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
0
votes
1answer
63 views

Why lasso yield a higher mse then ridge?

I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction ...
4
votes
0answers
71 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
1
vote
0answers
57 views

Relationship between regularization parameter in Ridge/Lasso with budget constraint

The equation for lasso and ridge regression are given as follows in the ISLR textbook: The dual form of the above equations are given in terms of budget as below: I am wondering if there is a ...
1
vote
0answers
247 views

Glmnet: How to select Lambda and Alpha

I'd like to pick the optimal lambda and alpha using the Glmnet package. I'm open to all models (Ridge, Lasso, Elastic). I'm assuming some out of sample error/cross validation is the best model ...
2
votes
1answer
156 views

Why weights are not negative in Lasso regression?

I can understand that lasso could make some weights to zero and prevent over-fitting. But for all the figure I see about lasso regression, weight will stay at zero once it reach zero and increasing ...
0
votes
0answers
44 views

Why is train MSE bigger then test MSE?

I did a lasso and ridge regression. In both cases my train mse is bigger then my test mse and i ask myself why ?
0
votes
0answers
18 views

Bigger K for Cross validation ridge regression is better?

I do a ridge Regression and choose my Lambda via cross validation. When i set k = 10 in cross validation allways get smaller mse on test and training data set comparing to set k = 5. What is the ...
2
votes
1answer
75 views

Performance of Ridge and Lasso Regression depend on set.seed?

I try to do a ridge and lasso regression for out of sample predictions. The optimal lambda is chosed via cross validation. I run my results for different seeds in R. And depending on the seed i get a ...
1
vote
0answers
44 views

Ridge and Lasso Regression: Should I drop one reference category like in OLS? [duplicate]

I do a ridge and lasso regression with a data set that have categorial variables. Should i drop 1 reference category like in OLS or is it okay to run the regression with als categories as dummy ...
0
votes
0answers
25 views

Strange MSE as result in ridge and lasso regression

I did a lasso and ridge regression. In my data set i had p > n ( more variables then observations) . At the beginning of my analysis i had only 13 explenatory variables where some of the variables ...
0
votes
0answers
53 views

Choose optimal lambda sequence for cross validation

I do cross validation for rigde regression to find the optimal lambda and ask myself which sequence of lambda i should put in the cv.glmnet function. I tried different seqeunces and get different ...
2
votes
1answer
250 views

What are the implications of scaling the features to xgboost?

Doing research about the xgboost algorithm I went through the documentation. I have heard that xgboost does not care much about the scale of the input features In this approach trees are ...
2
votes
1answer
72 views

What modeling problem does ridge regression solve?

If your modeling problem is that you have too many features, a solution to this problem is LASSO regularization. By forcing some feature coefficients to be zero, you remove them, thus reducing the ...
3
votes
4answers
141 views

Why don't we want to choose a big $\lambda$ in ridge regression?

The author in this video at minute 16:15 says that: we don't want to choose big $\lambda$ values becuase the coefficients will become very small and therefore they might not be accurately ...
0
votes
1answer
32 views

Help in understanding the ridge regression solution break down?

I tried to follow Jann Goschenhofer's answer here, but I don't understand How $x_i^T$ in $Criterion_{Ridge} = \sum_{i=1}^{n}(y_i-x_i^T\beta)^2 + \lambda \sum_{j=1}^p\beta_j^2$ became just $X$ without ...
0
votes
0answers
20 views

General guidance on selecting the modelling appraoch

I need some general guidance on choosing different modelling approaches out there. I am mainly investigation the effect of climate variables on crop yield and develop a model to predict yield based ...
4
votes
1answer
119 views

Ridge regression: penalizing weights corresponding to larger-scale features

In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function. On page 5, in the little section called "...
0
votes
0answers
22 views

Why i bias is not so bad for out of sample predictions?

Lets say i want to do a out of sample prediction with lasso or ridge regression. Both models produce coefficents that are biased but have a smaller variance the the ols coeffients for example. Why ...
1
vote
0answers
22 views

Ridge regression algorithm [duplicate]

Could someone explain how ridge regression algorithm works, step by step? Without focusing too much on formulas but rather how the mechanism works.
3
votes
3answers
112 views

Why shrinkage estimators?

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction....