Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
787
questions
0
votes
0
answers
500
views
interview question: ridge regression the out-of-sample performance never change when tune the hyperparameter?
I happened to an interview question:
In Ridge regression, what does it imply if the out-of-sample performance never change however we tune the hyperparameter (the coefficient of L2 regularization)?
...
- 1,325
0
votes
0
answers
266
views
Interpretation of Elastic net having too low or high value of alpha
Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9)
My ...
1
vote
1
answer
98
views
Will we use ridge in linear regression if there is no multicolinearity
I know that adding L2 regularization (ridge) can reduce multicolinearity in linear regression. I originally understand as multicolinearity will increase the estimation variance and L2 regularization ...
- 1,325
3
votes
1
answer
284
views
LASSO vs AIC for submodel selection via nonzero coefficient variable selection
Suppose you have a linear model which you believe has too many variables -- a cubic in 10 lags, for example. You believe, without being certain, that it is probably quadratic, and maybe linear, and ...
- 3,117
2
votes
0
answers
48
views
Can we estimate independent parameters when $p > n$?
I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals.
I know that when we estimate fixed effects, the number of ...
- 63
0
votes
2
answers
271
views
Orthogonality of columns of the augmented design matrix for ridge regression
In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
- 109
2
votes
1
answer
930
views
Ridge and Lasso in GLMs
In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others ...
- 298
2
votes
1
answer
430
views
Why does ridge regression only have one hyperparameter $\lambda$?
Ridge Regression objective$$\underset{\beta}{\text{min}}
\sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$
SVM primal problem:
$$\begin{align}
\max_{\mathbf{\alpha}} \quad &\min_{\...
user318514
6
votes
1
answer
628
views
When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?
Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
- 177
3
votes
1
answer
117
views
Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?
I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
- 434
0
votes
0
answers
153
views
Why does area under curve not change from 0.5?
I have performed a ridge logistic regression with glmnet and now I look at the performance metric AUC. The script is:
...
- 528
1
vote
1
answer
230
views
How to optimise penalty parameter in ridge regression using AIC
So I know for a ridge regression model, we need to find an optimal $\lambda$ value.
I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
- 13
1
vote
1
answer
277
views
LassoCV vs RidgeCV in Python -- why are their default number of folds different?
In https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html, it says that LassoCV defaults to 5 folds.
In https://scikit-learn.org/stable/...
- 343
1
vote
0
answers
118
views
How do the training and cross validation mean squared error curves behave as a function of $\lambda$?
I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
- 604
5
votes
2
answers
735
views
Bias variance tradeoff of ridge regression with independent but non identically distributed error?
I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather ...
user136768
0
votes
0
answers
1k
views
R: Plot of the relationship between lambda values and coefficients in ridge regression
I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients:
...
- 111
14
votes
3
answers
4k
views
Why the Ridge Regression is NOT scale-invariant?
In the Element of Statistical Learning, Chapter 3, we know that the linear regression is scale-invariant since the scale matrix for coefficient will be canceled eventually, but the Ridge regression ...
- 265
1
vote
0
answers
63
views
How to prove that optimal solution for ridge regression can be expressed in a following form? [closed]
We know that for ridge regression:
$$\beta = (X^TX + \lambda I)^{-1}X^Ty.$$
How can I prove that the optimal solution for beta can be expressed as $$\beta = X^T*V.$$ for some V
- 71
2
votes
1
answer
410
views
Ridge regression and distribution of estimate?
When OLS overfits observed data, does it give skewed distribution of estimates?
- 21
1
vote
0
answers
31
views
What is the recommend function for Ridge regression [duplicate]
The following question is an answer for why lm.ridge and glmnet results are different and how to solve that. My question is ...
- 2,089
2
votes
0
answers
62
views
Effect of L2 regularization on Linear Regression
I am starting with L2 regularization on linear regression.
Case without regularization:
objective function $(Xw-y)^T(Xw-y)$
parameters vector $w= (X^TX)^{-1}X^Ty.$
Case with regularization:
...
- 31
0
votes
1
answer
110
views
How to choose the correct dataset transformation
I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction ...
- 1
2
votes
0
answers
218
views
How does ridge regression actually utilize principal components?
This is what I understand so far:
Let $\mathbf X$ be the centered $n \times p$ predictor matrix and
consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a ...
- 309
2
votes
0
answers
146
views
Does the test error for ridge regression include the regularization term or not? [closed]
When you compute the test error for ridge regression, is it typically computed with the regularization term in it?
- 343
0
votes
0
answers
415
views
Cross validation for kernel regression
I've been reading about the Kernel trick, where we we can obtain a prediction by calculating: $\hat{y} = y(K(x,x)+\lambda I_n)^{−1} K(x,\hat{x})$, where $K(x,z) = (1+xz)^p$. If I want to tune the ...
- 1
2
votes
0
answers
112
views
Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?
I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
1
vote
0
answers
74
views
Can regularisation reduce the accuracy in the validation test?
I am constructing a CNN neural network with TensorFlow. I have run two versions of the CNN, one of them without regularization and the other with a kernel regularizer $L^2$ in each convolutional layer....
- 21
1
vote
0
answers
23
views
Weight regularisation in CNN
I am trying to understand the concept of weight regularisation in CNN. I know that in dense layer with weight $w$ it corresponds to finding:
$$
\mathbf{w}^{*}=\underset{\mathbf{w}}{\arg \operatorname{...
- 41
2
votes
1
answer
753
views
Intuitive understanding and practice of Group Lasso
I have been searching for a straight answer to clarify myself about the use of dummy variables in Lasso regression. I understand why we need to group them but I could not find any clear information on ...
- 31
2
votes
0
answers
1k
views
Why Ridge and Lasso regression is returning almost identical results to Linear Regression [closed]
I was trying to compare Ridge, Lasso and Linear Regression models to each other. I am using a subset of the Ames housing dataset. Here is a link to an already preprocessed dataset that I am using. The ...
- 121
5
votes
1
answer
1k
views
How is the standard error calculated for the `lambda.1se` output in the cv.glmnet function?
I understand that lambda.1se is the largest lambda that gives MSE within one standard error of the minimum MSE. But how is the standard error calculated exactly.
- 349
0
votes
0
answers
87
views
Why wouldn't you perform PCA before performing ridge regression on highly correlated parameters?
I'm trying to wrap my head around the L2 regularization component in ridge regression, to build a model on noisy, correlated data.
I understand the $\lambda$ introduces a penalty for high bias during ...
- 173
1
vote
1
answer
136
views
Numerical solution to the constrained ridge regression
The constrained ridge regression problem is of the form:
$\arg\min_{\|\beta\|_2\le t}\|X\beta-y\|_2$. Given a matrix $X$, a vector $y$ and the constrain parameter $t$, how do you solve it numerically?
...
- 11
1
vote
1
answer
124
views
Bias of Ridge Estimates in Regression
For a given ridge parameter, ridge estimates minimize the sum of squared predictions subject to an inequality constraint.
Are the ridge estimates biased regardless of whether the aforementioned ...
- 133
12
votes
1
answer
5k
views
Why regularization parameter called as lambda in theory and alpha in python?
I was learning about regularization and came across the term called regularization parameter.
I see that it is called lambda in ...
- 3,272
0
votes
1
answer
828
views
Why feature selection using `L1` and not using `L2` norm? [duplicate]
I read a tutorial here. In which, I came across the below plots
I read an explanation quoted ...
- 3,272
3
votes
0
answers
1k
views
L1 vs L2 norm - Circle and Diamond [duplicate]
I am new to ML and recently came across the L1 and L2 norm.
The tutorials that I read here and here show some circle and diamond ...
- 3,272
6
votes
0
answers
248
views
Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?
I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I ...
- 161
2
votes
1
answer
236
views
In which cases should I split the data in training set and test set [closed]
I am taking a course on machine learning and in one problem I should perform a Ridge regression to fit some given data to a known model. I was wondering if, in this case, there are any advantage in ...
- 21
3
votes
1
answer
94
views
On a Ridge regression-like problem
I am trying to implement a sort of Ridge regression for the following problem
\begin{equation}
y = a^\top X b,
\end{equation}
where $y\in \mathbb{R}$, $a\in\mathbb{R}^M$, $b\in\mathbb{R}^N$ and $X\in\...
- 33
2
votes
3
answers
5k
views
How to select the best feature set from Ridge regression?
I have applied L2 regularization on my features and have got coefficient values as below(hiding the column name due to client work:
I am unsure about what all features should I choose? Should I ...
- 111
2
votes
0
answers
303
views
Intuition behind the degrees of freedom in ridge regression
I'm reading through the ESL book and I'm on the part of ridge regression where the effective degrees of freedom are defined
$$
df(\lambda) = tr(X(X'X + \lambda I)^{-1}X') = \sum_{j=1}^p{\frac{d_j^2}{...
- 141
0
votes
1
answer
55
views
Ridge Regression - Advice on Modeling Sales Data
I am looking to use ridge regression to predict end of quarter sales revenue. My features are sales pipeline and revenue booked quarter to date. As the quarter progresses sales pipeline will ...
- 3
2
votes
2
answers
857
views
Why Weighted Ridge Regression gives same results as weighted least squares only when solved iteratively?
I was experimenting with weighted ridge regression for a linear system, where the closed-form solution is given by:
$$
b =(X^T WX + \lambda I)^{-1}X^T W y
$$
and also weighted least squares whose ...
- 21
0
votes
1
answer
753
views
Can we use Gradient Descent in the place of Ridge Regression in overfitting problem while doing linear regression problem?
What is the difference between Gradient Descent and Ridge regression?
We use ridge regression for overfitting problem when the Mean Squared Error for test dataset is high. I think that we can use ...
5
votes
2
answers
154
views
Given two equations that differ by one predictor, under ridge regression, which estimates are generally larger in magnitude?
Suppose we have two equations
$$
Y=\alpha_1X_1+\alpha_3X_3
$$
and
$$
Y=\beta_1X_1+\beta_2X_2+\beta_3X_3
$$
Suppose further that $X_1=X_2$, then would it usually be the case that $\hat{\alpha_1}$ or $\...
- 4,524
2
votes
1
answer
783
views
norm of ridge regression estimator
is there a characterization or an upper bound on the norm of the ridge regression estimator (coefficients)? As the Tikhonov regularization attempts to regularize these coefficients as part of the ...
- 2,545
1
vote
1
answer
301
views
L2 regularization: large value of coefficient <=> large variance <=> overfitting
For the loss function with L2 regularization:
$$Loss\ function + \lambda||w||^2_2.$$
I think following three things are equivalent with large probability:
large estimation value of $w^i$ <=> ...
- 1,325
2
votes
1
answer
576
views
Properties of ridge regression hat matrix and ridge residuals
I'm referencing https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 34 question 1.5 we need to prove :
Ridge fit $\widehat{Y}(\lambda)=X(X^{\top}X+\lambda I_p)^{-1}X^{\top}Y$ is not ...
- 23
1
vote
0
answers
202
views
How to make the regressor of LASSO consistent?
Suppose there is one regressor $X$ ith a sample so that $\sum_{i=1}^n X_i^2=n$. And suppose the OLS estimator of $Y$ on $X$ (no intercept) is consistent. What condition does $\lambda$ need to satisfy ...
- 23