Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
683
questions
0
votes
1answer
24 views
How to obtain odds ratio (and 95% CI) from ridge regression model
I am currently working on a ridge logistic (predictive) model. I was able to complete most of the steps and obtain the coefficient but I keep getting an error message when it comes to the odds ratio &...
0
votes
0answers
16 views
Ridge regression - prove derivate is zero at Q* [closed]
How to I prove that the optimal point in L2 ridge regression will make the derivative 0? It's kind of reverse work( I've substituted the q* value in the gradient but not able to prove its 0.
L(Q) = 1/...
0
votes
0answers
30 views
Limiting behavior of the ridge regression estimator as $\lambda \to \infty$
I am a bit confused about a few aspects of the behavior of the ridge regression estimator as $\lambda \to \infty$ (see photos below).
The facts that the bias is $- \lambda (\mathbf X^\top \mathbf X + \...
1
vote
0answers
34 views
Why do linear bandits use ridge regression to estimate parameters?
I’m implementing an adaptive experimental design where arms are assigned according to the posterior probability that they are the best arm. I’ve noticed in several articles that people use ridge ...
0
votes
0answers
17 views
Framework for applying weights to binary variables in regression
Say I am training a ridge regression model on nothing but binary variables. The context being that each variable represents a player - a value of 1 meaning they were playing the game at the time, ...
1
vote
1answer
37 views
Non-Ridge Kernelized Regression?
Every presentation that I have seen for kernelized regression focuses on finding
$$\underset{f \in \mathcal{H}_k}{\min} \sum_{i=1}^{n}(y_i-f(\mathbf{x}_i))^2+\lambda \|f\|^2_{\mathcal{H}_k}.$$
Here, $\...
2
votes
1answer
28 views
Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?
According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as:
...
0
votes
0answers
40 views
thresholding prior to model evaluation
Methodology question. The ML textbook approach is this:
perform model fit - optimisation
assess fit with Cross-Validation
tune decision rule by thresholding on the prediction probability (...
2
votes
1answer
51 views
Variance of the ridge regression estimator
I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$):
My main concern is that this derivation of ...
1
vote
1answer
29 views
Expected value of the ridge regression estimator
I am trying to understand this derivation:
I think everything except the last equality is fairly simple, but I do not understand the last equality. Is there an error here?
I appreciate any help.
0
votes
1answer
13 views
Lasso and SGDRegressor are not working well
I want to fit some data using Lasso, Ridge and SGDRegressor and to compare the results.
...
2
votes
1answer
33 views
Bayesian Elastic Lasso
While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need ...
0
votes
0answers
6 views
Appropriate regression framework for evaluating best players
I would like to model the best performers in a game using a ridge regression approach similar to RAPM in the NBA.
Background: The game involves two teams (say team A and team B) of five players each. ...
0
votes
0answers
27 views
Why does the ridge penalty shrink the singular values? [duplicate]
I am trying to understand the following analysis of ridge regression.
I am new to SVD but I think I have a sufficient grasp on most of the content. There are two things I am struggling with.
The ...
0
votes
0answers
20 views
In Ridge/Lasso Regression, What's The Advantage To Using CV Lamda And Then Some Form Of Training/Testing
When running a lasso or ridge regression, cross-validation allows us to find an optimal (minimized lamda.)
So - if we were using glmnet with a logistic response variable...
...
2
votes
1answer
40 views
Is Individual Coefficient Significance with Ridge or Lasso possible, when Amount of Variables exceeds Observations
First, to introduce you to my situation, I have a dataset containing n = 16 observations and p = 17 variables. My variable set contains 16 independent variables (14 variables I'm interested in and two ...
1
vote
1answer
25 views
Sign change in LASSO and RIDGE of coefficients
I am estimating in total three models: Logistic regression without any penalization (as benchmark model), logistic regression with L1 penalization (LASSO) and with L2 penalization (RIDGE). Now i ...
0
votes
0answers
19 views
Is it possible (reasonable) to weight the regularization for some variable in Ridge/elastic net based on their importance/causal effect
Say I have 100 predictor variables. And I have estimations from a causal inference method that indicates the causal effect size of each variable to the response variable.
Then I want to build a linear ...
3
votes
0answers
27 views
Proof of invariant angle between $Y$ and $\hat Y$ in $L^2$ regularisation
On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that ...
0
votes
1answer
140 views
Lasso vs Ridge Regression
My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). ...
0
votes
1answer
32 views
Minimizing $L_2$ norm with constrained residual sum of squares (RSS)
I have some complex-valued time-series data, $y \in \mathbb{C}^n$ - a signal with additive Gaussian white noise. The goal is to find the Fourier coefficients of this signal.
Ideally, you would just do ...
0
votes
1answer
33 views
How exactly does the glmnet in R determine the penalty in ridge regression?
in R, once I call https://www.rdocumentation.org/packages/glmnet/versions/4.1-2/topics/cv.glmnet with alpha = 0, I will magically get a set of coefficients from ridge regression, without having to ...
1
vote
1answer
91 views
Newton's method for Bernouilli likelihood with ridge penalty
I am trying to derive the gradient and hessian of logistic regression with ridge penalty. The log-likelihood should be (correct me if I am wrong):
$$\sum_{i=0}^n\Big(\log{(P_i^{y_i}(1-P_i)^{1-y_i}- \...
0
votes
0answers
86 views
Rule of thumb suggestion for Spark's Logistic Regression regParam and elasticNetParam?
Spark Logistic Regression page mentions the hyper parameters regParam and elasticNetParam both with defaults of zero.
Perhaps you may have some insight on the regParam and elasticNetParam rule of ...
0
votes
0answers
6 views
when we say correlation is 1 in boss Ridge and Elastic Net, does it only mean $x_1 = x_2$ for the allocation of weights
Question:
when we say correlation is 1 in boss Ridge and Elastic Net, does it only mean $x_1 = x_2?$
Story:
Ridge will trends to allocate the similar coefficients ...
2
votes
0answers
22 views
Compare the MSE between LASSO and OLS
In regression, the MSE of estimation $\hat{\theta}$ is:
$$MSE = E[(\hat{\theta} - \theta)^2].$$
I know the detailed comparison of MSE between OLS and Ridge. But can hardly find some materials between ...
2
votes
1answer
41 views
Rewriting the Ridge Regression coefficients
In Ridge Regression we try to find the minimum of the following loss function:
$$\text{min}_w\mathcal{L}_{\lambda}(w,S)=\text{min}\lambda\|w\|^2+\sum^l_{i=1}(y_i-g(x_i))^2$$
Where:
$\lambda$ is a ...
0
votes
1answer
79 views
Variable with negative coefficient in Ridge Regression and positive correlation
In my research, I have four independent variables (X1, X2, X3 and ...
1
vote
1answer
29 views
Lollipop plot in R with Ridge Regression coefficients [closed]
I have the following dataset, in which I want to understand the influence of four explanatory variables (X1, X2, ...
0
votes
1answer
61 views
Ridge Regression Graph
I intend to assess how some variables relate to a particular response variable:
...
0
votes
1answer
109 views
What is sparse model?
I am currently learning about Ridge and Lasso Regression, which leads me to learn about L1 and L2 regularization. There's a phrase saying ...
0
votes
0answers
97 views
interview question: ridge regression the out-of-sample performance never change when tune the hyperparameter?
I happened to an interview question:
In Ridge regression, what does it imply if the out-of-sample performance never change however we tune the hyperparameter (the coefficient of L2 regularization)?
...
0
votes
0answers
53 views
Interpretation of Elastic net having too low or high value of alpha
Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9)
My ...
0
votes
0answers
167 views
K-fold Cross Validation for ridge regression model evaluation with specific lambda value in R
I have identified the optimal lambda for a ridge regression model using k-fold cross validation.
However now I want to use k-fold cross validation to evaluate the model performance on different ...
0
votes
1answer
37 views
Will we use ridge in linear regression if there is no multicolinearity
I know that adding L2 regularization (ridge) can reduce multicolinearity in linear regression. I originally understand as multicolinearity will increase the estimation variance and L2 regularization ...
2
votes
1answer
58 views
LASSO vs AIC for submodel selection via nonzero coefficient variable selection
Suppose you have a linear model which you believe has too many variables -- a cubic in 10 lags, for example. You believe, without being certain, that it is probably quadratic, and maybe linear, and ...
0
votes
0answers
39 views
When to use ridge regression or lasso rather than elastic net?
If one has no ex-ante information about what the L1-ratio hyperparameter should be in the context of elastic net regularization, when should one instead use lasso or ridge? This ratio is referred to ...
2
votes
0answers
44 views
Can we estimate independent parameters when $p > n$?
I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals.
I know that when we estimate fixed effects, the number of ...
0
votes
2answers
144 views
Orthogonality of columns of the augmented design matrix for ridge regression
In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
1
vote
1answer
54 views
Ridge and Lasso in GLMs
In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others ...
0
votes
0answers
31 views
Inner Product RKHS and regression
Say you want to solve the kernel ridge regression as follows:
$\min\limits_{f\in H_K} \left[\lambda||f||^{2}_{H_K} + \sum\limits_{i=1}^{N}(y_i - f(x_i))^2\right]$, where $\lambda > 0$.
We know how ...
2
votes
1answer
90 views
Why does ridge regression only have one hyperparameter $\lambda$?
Ridge Regression objective$$\underset{\beta}{\text{min}}
\sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$
SVM primal problem:
$$\begin{align}
\max_{\mathbf{\alpha}} \quad &\min_{\...
6
votes
1answer
197 views
When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?
Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
2
votes
1answer
87 views
Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?
I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
0
votes
0answers
69 views
Why does area under curve not change from 0.5?
I have performed a ridge logistic regression with glmnet and now I look at the performance metric AUC. The script is:
...
1
vote
1answer
45 views
How to optimise penalty parameter in ridge regression using AIC
So I know for a ridge regression model, we need to find an optimal $\lambda$ value.
I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
1
vote
1answer
83 views
LassoCV vs RidgeCV in Python -- why are their default number of folds different?
In https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html, it says that LassoCV defaults to 5 folds.
In https://scikit-learn.org/stable/...
1
vote
0answers
19 views
How do the training and cross validation mean squared error curves behave as a function of $\lambda$?
I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
2
votes
2answers
165 views
Bias variance tradeoff of ridge regression with independent but non identically distributed error?
I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather ...
0
votes
0answers
55 views
R: Plot of the relationship between lambda values and coefficients in ridge regression
I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients:
...
