Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Interpreting Shapley Values on Breast Cancer

I was analyzing Shapley Values on the Wisconsin breast cancer data set (binary classification). I applied it on Random Forest and on Ridge and Lasso Regression. However the summary plot seems to be ...
user avatar
  • 1
5 votes
1 answer
60 views

Number of samples in scikit-Learn cost function for Ridge/Lasso regression

I am using scikit-learn to train some regression models on data and noticed that the cost function for Lasso Regression is defined like this: , whereas the cost function for e.g. Ridge Regression is ...
user avatar
0 votes
0 answers
17 views

Detailed comparison of two methods for obtaining the ridge regression solution

I have come across two different ways of obtaining the ridge regression solution, which are as follows: Method1:-(obtained from here) $RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
user avatar
  • 187
3 votes
0 answers
33 views

Ridge Regression Alpha/Lambda: Basic Characteristics?

I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (...
user avatar
6 votes
1 answer
222 views

Ridge or multiple linear regression following PCA?

I have a real world clinical dataset with a severe issue of p >> n. I have thus decided to run PCA before modelling the data. This leads to a dataset with 150 samples with 85 features. I would ...
user avatar
  • 185
0 votes
0 answers
32 views

Ridge Regression: Ridge traces are barely change across different k values [EDIT]

I am working with Matlab and following the example laid out here: https://www.mathworks.com/help/stats/ridge.html I then use the code above on my own data (with ~ 80 features). However, no matter what ...
user avatar
  • 185
1 vote
1 answer
28 views

Ridge classification: Interpreting prediction

I'm particularly concerned about the following problem when using ridge classification for predicting binary outcome When I'm encoding the binary outcome as 1 and 0; my model accuracy is 0.6456 When ...
user avatar
  • 31
1 vote
1 answer
58 views

Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge outputs correlations

Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge regression outputs are equal to the correlations used in correlation ...
user avatar
  • 31
1 vote
0 answers
43 views

How is Cholesky decomposition used in ridge regression?

As far as I learnt, Cholesky decomposition can be used only for symmetrical positive definite matrices, but I can see it is used as solver in Sklearn-Ridge package, can somebody explain how it is used ...
user avatar
  • 11
0 votes
3 answers
33 views

Why does test MSE always decrease with increasing training size (and decreasing test size)?

Context: I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
user avatar
0 votes
0 answers
21 views

Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
user avatar
  • 11
0 votes
1 answer
38 views

How to extract MSEP or RMSEP from lassoCV?

I'm doing lasso and ridge regression in R with the package chemometrics. With ridgeCV it is easy to extract the SEP and MSEP values by ...
user avatar
  • 21
1 vote
0 answers
27 views

Mixed model via ridge regression

A mixed model can be recast as a ridge regression for a specific regularization parameter $\lambda$ that penalizes only the random effects -- aka dummy variables for the grouping levels. Fitting a ...
user avatar
  • 12.1k
1 vote
2 answers
38 views

Introduction to Statistical Learning Eq. 6.12 and 6.13

Can someone please explain me how the optimization of 6.12 leads to 6.14 and the optimization of 6.13 leads to 6.15?
user avatar
  • 93
0 votes
1 answer
20 views

Ridge regression coefficients show model importance but the model evaulation not

I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
user avatar
  • 332
1 vote
0 answers
27 views

What is the consequence of "copying" a dataset for Ridge or Lasso?

So I know that for OLS, "copying" each of the N observations $(X_i,Y_i)$ once to get a dataset of size 2N has no effect on the values of the coefficients in OLS (related question). Does this ...
user avatar
  • 101
6 votes
2 answers
557 views

Lasso coefficient for some features is higher than Linear Regression Coefficient

I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, ...
user avatar
0 votes
1 answer
215 views

Is Ridge more robust than Lasso on feature selection?

My goal is to identify the best n-feature linear model, i.e. pick the model with only n-feature from total N features (n < N) and lowest Mean-Squared-Error (MSE). The experiment is on the Lasso and ...
user avatar
  • 63
0 votes
0 answers
77 views

Normalization and RidgeCV in Sklearn Pipeline - possible data leakage?

To avoid data leakage between the train and test set, I'm using sklearn's Pipeline as follows: ...
user avatar
2 votes
1 answer
139 views

What would be the exact function for a ridge logistic regression with multiple variables?

I am looking for the correct equation for a ridge logistic regression for multiple variables. I thought it simply was: $$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}$$ with an ...
user avatar
  • 332
0 votes
1 answer
47 views

How to "choose" binary variables which have a big impact on a regression?

I am currently facing an issue with analyzing my data for a project. I have a dataset of about 100.000 samples. I have approximate 50 columns which are all binary and my dependent variable is time ...
user avatar
0 votes
0 answers
28 views

In ridge regression, Why choose regression vector which has a minimum length?

As I reading a thesis named 'Ridge Regression: Biased Estimation for Nonorthogonal Problem' written by Hoerl and Kennard, I was struck by the below problem. Let $\boldsymbol{B}$ be any estimate of the ...
user avatar
1 vote
0 answers
61 views

Standard Error of Ridge regression

Why is it advised to use bootstrap for finding the SE for the ridge regression estimator? Using the above formula, we can get the SE much faster than bootstrap. But all the research papers in ...
user avatar
  • 11
5 votes
1 answer
173 views

Ridge regression subtlety on intercept

I just noticed that when using ridge regression, there is a small subtlety on the penalised parameters, namely, we don't penalise $\theta_0$. Can someone give me a simple and intuitive explanation of ...
user avatar
  • 113
3 votes
3 answers
283 views

Why RidgeClassifier can be significantly faster than LogisticRegression with a high number of classes?

In Scikit document, we can find this statement The RidgeClassifier can be significantly faster than e.g. LogisticRegression with a high number of classes because it can compute the projection matrix $...
user avatar
  • 247
2 votes
1 answer
265 views

Ridge Trace Plot - Interpretation

In my research, I aimed to perform a regression model with four predictors and one response variable. When I verified a high collinearity among the predictors, I was instructed to handle this problem ...
user avatar
  • 47
1 vote
0 answers
114 views

Feature Selection via RFE with Ridge or SVM (Regression)

I have a regression problem where the number of samples n is less than the number of features p (e.g., p=500 and n=400; but the problem can be extended to p=3000+ and n=400). The features are largely ...
user avatar
  • 13
2 votes
2 answers
60 views

Ridge regression with shrinkage towards nonzero matrix

Suppose I want to perform ridge-regularized linear regression, except that we shrink the coefficients to a nonzero matrix: $$ W^* = \arg\min_W \|Y - X W \|^2_2 + \lambda\|W-W_0\|^2_2. $$ However, I ...
user avatar
0 votes
1 answer
92 views

How to obtain odds ratio (and 95% CI) from ridge regression model

I am currently working on a ridge logistic (predictive) model. I was able to complete most of the steps and obtain the coefficient but I keep getting an error message when it comes to the odds ratio &...
user avatar
0 votes
0 answers
58 views

Limiting behavior of the ridge regression estimator as $\lambda \to \infty$

I am a bit confused about a few aspects of the behavior of the ridge regression estimator as $\lambda \to \infty$ (see photos below). The facts that the bias is $- \lambda (\mathbf X^\top \mathbf X + \...
user avatar
  • 531
1 vote
0 answers
46 views

Why do linear bandits use ridge regression to estimate parameters?

I’m implementing an adaptive experimental design where arms are assigned according to the posterior probability that they are the best arm. I’ve noticed in several articles that people use ridge ...
user avatar
  • 11
0 votes
0 answers
30 views

Framework for applying weights to binary variables in regression

Say I am training a ridge regression model on nothing but binary variables. The context being that each variable represents a player - a value of 1 meaning they were playing the game at the time, ...
user avatar
1 vote
1 answer
57 views

Non-Ridge Kernelized Regression?

Every presentation that I have seen for kernelized regression focuses on finding $$\underset{f \in \mathcal{H}_k}{\min} \sum_{i=1}^{n}(y_i-f(\mathbf{x}_i))^2+\lambda \|f\|^2_{\mathcal{H}_k}.$$ Here, $\...
user avatar
2 votes
1 answer
102 views

Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?

According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as: ...
user avatar
0 votes
0 answers
50 views

thresholding prior to model evaluation

Methodology question. The ML textbook approach is this: perform model fit - optimisation assess fit with Cross-Validation tune decision rule by thresholding on the prediction probability (...
user avatar
  • 1
2 votes
1 answer
195 views

Variance of the ridge regression estimator

I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$): My main concern is that this derivation of ...
user avatar
  • 531
1 vote
1 answer
89 views

Expected value of the ridge regression estimator

I am trying to understand this derivation: I think everything except the last equality is fairly simple, but I do not understand the last equality. Is there an error here? I appreciate any help.
user avatar
  • 531
0 votes
1 answer
83 views

Lasso and SGDRegressor are not working well

I want to fit some data using Lasso, Ridge and SGDRegressor and to compare the results. ...
user avatar
2 votes
1 answer
37 views

Bayesian Elastic Lasso

While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need ...
user avatar
  • 2,585
0 votes
0 answers
7 views

Appropriate regression framework for evaluating best players

I would like to model the best performers in a game using a ridge regression approach similar to RAPM in the NBA. Background: The game involves two teams (say team A and team B) of five players each. ...
user avatar
0 votes
0 answers
32 views

Why does the ridge penalty shrink the singular values? [duplicate]

I am trying to understand the following analysis of ridge regression. I am new to SVD but I think I have a sufficient grasp on most of the content. There are two things I am struggling with. The ...
user avatar
  • 531
0 votes
0 answers
36 views

In Ridge/Lasso Regression, What's The Advantage To Using CV Lamda And Then Some Form Of Training/Testing

When running a lasso or ridge regression, cross-validation allows us to find an optimal (minimized lamda.) So - if we were using glmnet with a logistic response variable... ...
user avatar
2 votes
1 answer
65 views

Is Individual Coefficient Significance with Ridge or Lasso possible, when Amount of Variables exceeds Observations

First, to introduce you to my situation, I have a dataset containing n = 16 observations and p = 17 variables. My variable set contains 16 independent variables (14 variables I'm interested in and two ...
user avatar
1 vote
1 answer
45 views

Sign change in LASSO and RIDGE of coefficients

I am estimating in total three models: Logistic regression without any penalization (as benchmark model), logistic regression with L1 penalization (LASSO) and with L2 penalization (RIDGE). Now i ...
user avatar
  • 27
0 votes
0 answers
33 views

Is it possible (reasonable) to weight the regularization for some variable in Ridge/elastic net based on their importance/causal effect

Say I have 100 predictor variables. And I have estimations from a causal inference method that indicates the causal effect size of each variable to the response variable. Then I want to build a linear ...
user avatar
4 votes
0 answers
35 views

Proof of invariant angle between $Y$ and $\hat Y$ in $L^2$ regularisation

On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that ...
user avatar
  • 315
0 votes
1 answer
319 views

Lasso vs Ridge Regression

My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). ...
user avatar
  • 27
0 votes
1 answer
41 views

Minimizing $L_2$ norm with constrained residual sum of squares (RSS)

I have some complex-valued time-series data, $y \in \mathbb{C}^n$ - a signal with additive Gaussian white noise. The goal is to find the Fourier coefficients of this signal. Ideally, you would just do ...
user avatar
  • 163
0 votes
1 answer
96 views

How exactly does the glmnet in R determine the penalty in ridge regression?

in R, once I call https://www.rdocumentation.org/packages/glmnet/versions/4.1-2/topics/cv.glmnet with alpha = 0, I will magically get a set of coefficients from ridge regression, without having to ...
user avatar
1 vote
1 answer
137 views

Newton's method for Bernouilli likelihood with ridge penalty

I am trying to derive the gradient and hessian of logistic regression with ridge penalty. The log-likelihood should be (correct me if I am wrong): $$\sum_{i=0}^n\Big(\log{(P_i^{y_i}(1-P_i)^{1-y_i}- \...
user avatar
  • 35

1
2 3 4 5
14