Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
0
votes
1answer
24 views

How to obtain odds ratio (and 95% CI) from ridge regression model

I am currently working on a ridge logistic (predictive) model. I was able to complete most of the steps and obtain the coefficient but I keep getting an error message when it comes to the odds ratio &...
0
votes
0answers
16 views

Ridge regression - prove derivate is zero at Q* [closed]

How to I prove that the optimal point in L2 ridge regression will make the derivative 0? It's kind of reverse work( I've substituted the q* value in the gradient but not able to prove its 0. L(Q) = 1/...
0
votes
0answers
30 views

Limiting behavior of the ridge regression estimator as $\lambda \to \infty$

I am a bit confused about a few aspects of the behavior of the ridge regression estimator as $\lambda \to \infty$ (see photos below). The facts that the bias is $- \lambda (\mathbf X^\top \mathbf X + \...
1
vote
0answers
34 views

Why do linear bandits use ridge regression to estimate parameters?

I’m implementing an adaptive experimental design where arms are assigned according to the posterior probability that they are the best arm. I’ve noticed in several articles that people use ridge ...
0
votes
0answers
17 views

Framework for applying weights to binary variables in regression

Say I am training a ridge regression model on nothing but binary variables. The context being that each variable represents a player - a value of 1 meaning they were playing the game at the time, ...
1
vote
1answer
37 views

Non-Ridge Kernelized Regression?

Every presentation that I have seen for kernelized regression focuses on finding $$\underset{f \in \mathcal{H}_k}{\min} \sum_{i=1}^{n}(y_i-f(\mathbf{x}_i))^2+\lambda \|f\|^2_{\mathcal{H}_k}.$$ Here, $\...
2
votes
1answer
28 views

Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?

According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as: ...
0
votes
0answers
40 views

thresholding prior to model evaluation

Methodology question. The ML textbook approach is this: perform model fit - optimisation assess fit with Cross-Validation tune decision rule by thresholding on the prediction probability (...
2
votes
1answer
51 views

Variance of the ridge regression estimator

I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$): My main concern is that this derivation of ...
1
vote
1answer
29 views

Expected value of the ridge regression estimator

I am trying to understand this derivation: I think everything except the last equality is fairly simple, but I do not understand the last equality. Is there an error here? I appreciate any help.
0
votes
1answer
13 views

Lasso and SGDRegressor are not working well

I want to fit some data using Lasso, Ridge and SGDRegressor and to compare the results. ...
2
votes
1answer
33 views

Bayesian Elastic Lasso

While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need ...
0
votes
0answers
6 views

Appropriate regression framework for evaluating best players

I would like to model the best performers in a game using a ridge regression approach similar to RAPM in the NBA. Background: The game involves two teams (say team A and team B) of five players each. ...
0
votes
0answers
27 views

Why does the ridge penalty shrink the singular values? [duplicate]

I am trying to understand the following analysis of ridge regression. I am new to SVD but I think I have a sufficient grasp on most of the content. There are two things I am struggling with. The ...
0
votes
0answers
20 views

In Ridge/Lasso Regression, What's The Advantage To Using CV Lamda And Then Some Form Of Training/Testing

When running a lasso or ridge regression, cross-validation allows us to find an optimal (minimized lamda.) So - if we were using glmnet with a logistic response variable... ...
2
votes
1answer
40 views

Is Individual Coefficient Significance with Ridge or Lasso possible, when Amount of Variables exceeds Observations

First, to introduce you to my situation, I have a dataset containing n = 16 observations and p = 17 variables. My variable set contains 16 independent variables (14 variables I'm interested in and two ...
1
vote
1answer
25 views

Sign change in LASSO and RIDGE of coefficients

I am estimating in total three models: Logistic regression without any penalization (as benchmark model), logistic regression with L1 penalization (LASSO) and with L2 penalization (RIDGE). Now i ...
0
votes
0answers
19 views

Is it possible (reasonable) to weight the regularization for some variable in Ridge/elastic net based on their importance/causal effect

Say I have 100 predictor variables. And I have estimations from a causal inference method that indicates the causal effect size of each variable to the response variable. Then I want to build a linear ...
3
votes
0answers
27 views

Proof of invariant angle between $Y$ and $\hat Y$ in $L^2$ regularisation

On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that ...
0
votes
1answer
140 views

Lasso vs Ridge Regression

My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). ...
0
votes
1answer
32 views

Minimizing $L_2$ norm with constrained residual sum of squares (RSS)

I have some complex-valued time-series data, $y \in \mathbb{C}^n$ - a signal with additive Gaussian white noise. The goal is to find the Fourier coefficients of this signal. Ideally, you would just do ...
0
votes
1answer
33 views

How exactly does the glmnet in R determine the penalty in ridge regression?

in R, once I call https://www.rdocumentation.org/packages/glmnet/versions/4.1-2/topics/cv.glmnet with alpha = 0, I will magically get a set of coefficients from ridge regression, without having to ...
1
vote
1answer
91 views

Newton's method for Bernouilli likelihood with ridge penalty

I am trying to derive the gradient and hessian of logistic regression with ridge penalty. The log-likelihood should be (correct me if I am wrong): $$\sum_{i=0}^n\Big(\log{(P_i^{y_i}(1-P_i)^{1-y_i}- \...
0
votes
0answers
86 views

Rule of thumb suggestion for Spark's Logistic Regression regParam and elasticNetParam?

Spark Logistic Regression page mentions the hyper parameters regParam and elasticNetParam both with defaults of zero. Perhaps you may have some insight on the regParam and elasticNetParam rule of ...
0
votes
0answers
6 views

when we say correlation is 1 in boss Ridge and Elastic Net, does it only mean $x_1 = x_2$ for the allocation of weights

Question: when we say correlation is 1 in boss Ridge and Elastic Net, does it only mean $x_1 = x_2?$ Story: Ridge will trends to allocate the similar coefficients ...
2
votes
0answers
22 views

Compare the MSE between LASSO and OLS

In regression, the MSE of estimation $\hat{\theta}$ is: $$MSE = E[(\hat{\theta} - \theta)^2].$$ I know the detailed comparison of MSE between OLS and Ridge. But can hardly find some materials between ...
2
votes
1answer
41 views

Rewriting the Ridge Regression coefficients

In Ridge Regression we try to find the minimum of the following loss function: $$\text{min}_w\mathcal{L}_{\lambda}(w,S)=\text{min}\lambda\|w\|^2+\sum^l_{i=1}(y_i-g(x_i))^2$$ Where: $\lambda$ is a ...
0
votes
1answer
79 views

Variable with negative coefficient in Ridge Regression and positive correlation

In my research, I have four independent variables (X1, X2, X3 and ...
1
vote
1answer
29 views

Lollipop plot in R with Ridge Regression coefficients [closed]

I have the following dataset, in which I want to understand the influence of four explanatory variables (X1, X2, ...
0
votes
1answer
61 views

Ridge Regression Graph

I intend to assess how some variables relate to a particular response variable: ...
0
votes
1answer
109 views

What is sparse model?

I am currently learning about Ridge and Lasso Regression, which leads me to learn about L1 and L2 regularization. There's a phrase saying ...
0
votes
0answers
97 views

interview question: ridge regression the out-of-sample performance never change when tune the hyperparameter?

I happened to an interview question: In Ridge regression, what does it imply if the out-of-sample performance never change however we tune the hyperparameter (the coefficient of L2 regularization)? ...
0
votes
0answers
53 views

Interpretation of Elastic net having too low or high value of alpha

Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9) My ...
0
votes
0answers
167 views

K-fold Cross Validation for ridge regression model evaluation with specific lambda value in R

I have identified the optimal lambda for a ridge regression model using k-fold cross validation. However now I want to use k-fold cross validation to evaluate the model performance on different ...
0
votes
1answer
37 views

Will we use ridge in linear regression if there is no multicolinearity

I know that adding L2 regularization (ridge) can reduce multicolinearity in linear regression. I originally understand as multicolinearity will increase the estimation variance and L2 regularization ...
2
votes
1answer
58 views

LASSO vs AIC for submodel selection via nonzero coefficient variable selection

Suppose you have a linear model which you believe has too many variables -- a cubic in 10 lags, for example. You believe, without being certain, that it is probably quadratic, and maybe linear, and ...
0
votes
0answers
39 views

When to use ridge regression or lasso rather than elastic net?

If one has no ex-ante information about what the L1-ratio hyperparameter should be in the context of elastic net regularization, when should one instead use lasso or ridge? This ratio is referred to ...
2
votes
0answers
44 views

Can we estimate independent parameters when $p > n$?

I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals. I know that when we estimate fixed effects, the number of ...
0
votes
2answers
144 views

Orthogonality of columns of the augmented design matrix for ridge regression

In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
1
vote
1answer
54 views

Ridge and Lasso in GLMs

In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others ...
0
votes
0answers
31 views

Inner Product RKHS and regression

Say you want to solve the kernel ridge regression as follows: $\min\limits_{f\in H_K} \left[\lambda||f||^{2}_{H_K} + \sum\limits_{i=1}^{N}(y_i - f(x_i))^2\right]$, where $\lambda > 0$. We know how ...
2
votes
1answer
90 views

Why does ridge regression only have one hyperparameter $\lambda$?

Ridge Regression objective$$\underset{\beta}{\text{min}} \sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$ SVM primal problem: $$\begin{align} \max_{\mathbf{\alpha}} \quad &\min_{\...
6
votes
1answer
197 views

When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?

Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
2
votes
1answer
87 views

Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?

I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
0
votes
0answers
69 views

Why does area under curve not change from 0.5?

I have performed a ridge logistic regression with glmnet and now I look at the performance metric AUC. The script is: ...
1
vote
1answer
45 views

How to optimise penalty parameter in ridge regression using AIC

So I know for a ridge regression model, we need to find an optimal $\lambda$ value. I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
1
vote
1answer
83 views

LassoCV vs RidgeCV in Python -- why are their default number of folds different?

In https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html, it says that LassoCV defaults to 5 folds. In https://scikit-learn.org/stable/...
1
vote
0answers
19 views

How do the training and cross validation mean squared error curves behave as a function of $\lambda$?

I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
2
votes
2answers
165 views

Bias variance tradeoff of ridge regression with independent but non identically distributed error?

I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather ...
0
votes
0answers
55 views

R: Plot of the relationship between lambda values and coefficients in ridge regression

I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients: ...

1
2 3 4 5
14