Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
570
questions
0
votes
0answers
18 views
Penalizing non-OLS models
Let’s consider some common linear time series models for which OLS does not usually yield unbiased coefficient estimates. These include ARIMA and ARIMAX models, regression models with ARIMA errors, ...
3
votes
2answers
65 views
+100
Bayesian interpretation of logistic ridge regression
Most textbooks (also this blog) cover the fact that ridge regression,
$$
\hat y = \hat \beta X;
\\
\hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2}
+ \lambda \...
1
vote
1answer
26 views
How to implement the closed form solution of Ridge Regression in Python when intercept is not 0 (fit_intercept=True) without using sklearn?
The well-known closed-form solution of Ridge regression is:
I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no ...
1
vote
0answers
11 views
Mixed-effects models with lasso penalty and ridge penalty in R [closed]
I am using the PISA 2015 data and trying to run a mixed-effects ridge and lasso regression model Schools will be included as a random effect, and student-level (e.g. motivation, socioeconomic status, ...
3
votes
0answers
26 views
Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?
I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
1
vote
1answer
18 views
standardizing the MSE
I conducted a Ridge regression with k-fold validation. All the predictors were scaled prior to the regression procedure.
To report on the accuracy of my model's prediction, I calculated the MSE in the ...
2
votes
1answer
17 views
Misclassification for test and training sets
I have a problem where I need to provide the misclassification error for both training- and test-set. I am working with logistic regression, so I have a binomial family for my models. I have two ...
0
votes
0answers
13 views
Why Ridge regression doesn't depend on centering $y$ in sklearn?
In sklearn's manual for Ridge they wrote the following about its parameter "fit_intercept":
But it seems that Ridge model doesn't depend on whether $y$ is centered or not:
...
0
votes
0answers
21 views
Bias and variance calculation for Ridge estimator of β
I understand how bias and variance for ridge estimator of β are calculated when the model is Y=Xβ + ϵ. But I have the model Y=Xtβ + ϵ. I don't understand if a model like that makes sense, can someone ...
0
votes
0answers
12 views
Ridge regression on one predictor and the reduction of the risk of overfitting
In the book Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 1, the author stated that when doing Ridge Regression (for only one predictor), this regularization reduces the risk of ...
2
votes
1answer
44 views
How does multicollinearity affect the eigenvalues of a matrix?
I have been looking into ridge regression as a method to address multicollinearity in data.
I am aware that multicollinearity can cause high variance in coefficient estimates. I have seen equations ...
0
votes
1answer
27 views
In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random?
I am having a hard time understanding the Bayesian Ridge Regression.
In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random? or do we treat the $\lambda$ as given?...
0
votes
3answers
41 views
Relation between Frobenius norm and L2 norm?
Is there any relation between the Frobenius norm of a matrix and L2 norm of the vectors contained in this matrix.
Simply put, is there any difference between minimizing the Frobenius norm of a matrix ...
0
votes
2answers
31 views
How does the dimensions of my regression matrix effect its eigenvalules?
I was reading through another question answer where it discussed the inversion of the XX' matrix in ridge regression. It stated that it is impossible to have positive eigenvalues if matrix $XX'$(=$VDV'...
2
votes
0answers
26 views
Weird glmnet ridge regression results with an uncentered design matrix
I was recently trying to figure out what glmnet's ridge regression is doing (7,000 lines of Fortran are no fun) and am confused by its behavior with an uncentered design matrix $X$.
I am aware that ...
3
votes
1answer
48 views
l1 and l2 regularizations [duplicate]
What do the red ellipses mean to us in the chart below? Does it have a relationship with RMSE values?
Figure 3.11 from Elements of Statistical Learning by Hastie, Tibshirani, and Friedman
0
votes
1answer
27 views
Applying de-standardised ridge regression coefficients to new test data - how to best handle the mean of y_test?
this is my first post on Stackexchange, so please correct me in any way if I am doing it wrongly.
I just stumbled across this question, I was battling with the same issue, but the posts there ...
2
votes
1answer
67 views
Understanding Partial Likelihood Deviance vs lambda relationship in LASSO
I'm analyzing gene expression data using regularized linear regression models (lasso-elastic net-...
2
votes
1answer
70 views
Ridge Regression: how to show squared bias increases as $\lambda$ increases
I have a Ridge regression model to estimate the coefficients of the true model $y = X\beta + \epsilon$. I have the standard model where $\mathbb{E}[\epsilon] = 0, \ \mathrm{Var}(\epsilon) = I.$ The ...
1
vote
0answers
16 views
Should an elastic-net always outperform lasso?
Since the lasso is a subset of the elastic-net, shouldn't a continuous grid of the ridge and lasso paramaters in the elastic-net always outperform the lasso regression?
1
vote
1answer
27 views
How do I interpret the results of lasso and ridge regression?
I have created a ridge and lasso regression model in R. From my understanding, the coefficients are interpreted differently from logistic regression. For instance, in logistic regression you may say ...
2
votes
1answer
88 views
Trace of the Hat Matrix in Ridge Regression
Generally, I know that the trace of the hat matrix ($H$) is equal to the rank of H since it is an orthogonal projection. If I wanted to show the trace of $H$ in ridge regression, would I be able to ...
0
votes
0answers
8 views
Cross sectional dependence
I am working with a panel data set of neighbouring EU countries (n=9,t=24). The variables are CO2 emissions (dependent), GDP, Population and fossil fuel usage. I ran a ridge regression model on the ...
0
votes
1answer
44 views
identity matrix shape of ridge regression
I am trying to apply manually ridge regression for the b coefficients. So i am trying to make the inverse matrix from this formula: $\left( X^T X + \lambda I \right)^{-1}$, but I am not sure which ...
1
vote
1answer
33 views
ISLR - Ridge Regression - Demonstrate equal coefficients with correlated predictors? (final step help)
I am working through the ISLR book have made a solid effort at answering Question 5 (Chapter 6), but for some reason I am having some real difficulty wrapping my head around the final step. I really ...
0
votes
0answers
42 views
Ridge regression and low R-squared
I am currently working with a strongly balanced data set. I have 15 countries over the period 1990-2017. My dependent variable is CO2 emissions. My independent variables are GDP per capita, total ...
1
vote
2answers
54 views
Ridge regression and autocorrelation
Im currently working with a strongley balanced data set. I have 15 countries over the period 1990-2017. My dependent variable is CO2 emissions. My independent variables are as follows, GDP per capita, ...
1
vote
0answers
27 views
Why does Ridge Regression work better than LASSO in the presence of multicollinearity? [duplicate]
I am learning the Ridge Regression. And I found that Ridge Regression is better than LASSO to analyze multicollinear? I wonder the reason?
0
votes
0answers
27 views
Should GBlup method and Bayesian Ridge Regression give the same results?
I am working on genomic selection and I am comparing the performance of two models, one of them is a likelihood method (GBlup) and the other is a Bayesian meyhod (Bayesian Ridge Regression). I am ...
2
votes
0answers
17 views
Why we only use L1 and L2 in penalized regression [duplicate]
I understand we do not want to go lower than 1 because the problem is no longer convex. But why not use L3 for example? Is there any reason why L2 is so popular but not higher norm?
0
votes
1answer
18 views
Using prior knowledge about correlated variable in ridge regression
I am wondering what methods are available for incorporating prior knowledge of some variable that is correlated with the unknown regression coefficients in a ridge regression. I have a sparse matrix ...
1
vote
1answer
26 views
Relations between regularization constant and parameter space in ridge regularization
Can it be shown that the $||w^*||_2$, where $w^* = (X'X+\lambda I)^{-1}X'Y$ , is inversely affected by the regularization constant $\lambda$, i.e. it is of $O(\frac{1}{\lambda})$ ?
1
vote
1answer
28 views
identity matrix of ridge regression equation
I am trying to understand the equation of ridge regression but I have a hard time with it.
I would like to know what the identity matrix I of the equation below represents. I'm not sure what the ...
1
vote
0answers
32 views
Regularised Regression and Feature Scaling
When performing regularised regression, such as LASSO, ridge regression and elastic net, I understand that it is important to scale variables before calculating and applying a penalty term. I have ...
0
votes
1answer
72 views
Mixed Model: Ridge Regression and Data Augmentation
Supposed I have a mixed model in the form: $$y = X\beta + Zu+ \varepsilon$$
If I want to enforce a constraint on the $\beta$s can I follow the data augmentation approach that @whuber mentioned here: ...
2
votes
0answers
23 views
Understanding scaling coefficients in the Naive Elastic Net
So I was reading the original Elastic Net paper by Zou, Hastie and I got slightly confused in the second section, where the reduction from Elastic Net to Lasso is performed.
They propose that an ...
2
votes
1answer
30 views
I don't understand RidgeCV's fit_intercept, and how to use it for my data
Alright, I have an assignment that makes me calculate weights for a function with different terms. At first, I thought I might just leave the weight for the term $1$ out, and instead use the intercept....
2
votes
2answers
47 views
Least Square Fitted Vector Through SVD Equals to y
Elements of Statistical Learning, p 66
The SVD of the $N \times p$ matrix $X$ has the form
$X = UDV^T$
Here $U$ and $V$ are $N \times p$ and $p \times p$ orthogonal matrices, with the ...
0
votes
0answers
10 views
Understanding single outlier in model predictions of penalized regression model
I'd like to discuss the following prediction scenario of a fitted penalized (L2) regression model.
(The data is hosted statically on Zenodo and interactively downloaded to your temporary R directory -...
5
votes
1answer
52 views
Bayesian formulation of best subset regression
We know Ridge is equivalent to using a Gaussian prior and Lasso is equivalent to using a double exponential prior.
What is the Bayesian interpretation (implied prior) for the best subset regression? ...
3
votes
0answers
71 views
How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression?
How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression?
Given lasso (constrained form):
$$\underset{\beta}{\min}{(\frac{1}{2N}||y-x\beta||_2^2)} \...
5
votes
1answer
175 views
Lasso, Ridge and Best Subset estimator for orthogonal cases
I am reading the book "Elements of Statistical Learning". In the book the author compares the OSL estimator with Lasso, Ridge and Best Subset for the special case of Orthogonal X. I am attaching the ...
1
vote
0answers
19 views
For ridge regression, show if $K$ columns of $X$ are identical then we must have same corresponding parameters
Show if $K$ columns of $X$, $({X_{j1}, X_{j2}...X_{jk}}) $are identical then we must have $\hat\beta_{j1},\hat\beta_{j2},...\hat\beta_{jk} $ are same in the ridge regression:
$$\hat\beta = \underset{...
0
votes
0answers
8 views
What’s the difference between PQL regression and ridge or lasso or elastic net?
What’s the difference between the penalized methods, glmmPQL and elastic or ridge or elastic net?
0
votes
1answer
25 views
Is there a way to estimate ridge regression regularisation parameter from OLS output
When performing ridge regression there is a regularisation parameter to choose. Can a good approximate value be estimated from output from standard OLS.
Intuitively these factors seem important:
- ...
0
votes
1answer
29 views
Implementing a regularization term in ridge regrssion
I am trying to solve the ridge regression problem given by
$D_{s} = min_{D_{s}} \hspace{2mm} || X_{s} - D_{s}Y_{s}||_F^{2} + \lambda \hspace{2mm} ||D_{s}||_{F}^{2} \hspace{10mm} s.t. \hspace{5mm} ||...
2
votes
1answer
61 views
R: caret (elasticnet): ridge regression: understanding the returned parameters
I wanted to play around with the ridge regression in caret (which apparently uses elasticnet), so I did two experiments:
use the original data
use the modified data where the values of ...
0
votes
0answers
29 views
What is Ridge trace and Lasso path?
More specifically, what does "Ridge trace of regression coefficients" and "Lasso path of regression coefficients" mean? Are they the same as "Ridge coefficient paths" and "Lasso coefficient paths"?
2
votes
0answers
110 views
Why would one want to choose lambda.1se for ridge regression in glmnet?
In R, choosing lambda.1se over lambda.min to get a more parsimonious model is common. This post (and this) also indicated that ...
1
vote
1answer
57 views
Least Square vs Shrinkage approach of fitting models
What is the difference between the Least Square and Shrinkage approach of fitting models in the context of model selection?
In https://www.youtube.com/watch?v=QlyROnAjnEk the author at [0:28] instance ...
