Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
787
questions
9
votes
2
answers
3k
views
PRESS statistic for ridge regression
In ordinary least squares, regressing a target vector $y$ against a set of predictors $X$, the hat matrix is computed as
$$H = X (X^tX)^{-1} X^t$$
and the PRESS (predicted residual sum of squares) ...
- 3,682
9
votes
1
answer
572
views
Regularized fit from summarized data: choosing the parameter
Following on from my earlier question, the solution to the normal equations for ridge regression is given by:
$$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$
Could you offer any guidance for ...
- 5,591
9
votes
1
answer
2k
views
Range of lambda in elastic net regression
$\def\l{|\!|}$
Given the elastic net regression
$$\min_b \frac{1}{2}\l y - Xb \l^2 + \alpha\lambda \l b\l_2^2 + (1 - \alpha) \lambda \l b\l_1$$
how can an appropriate range of $\lambda$ be chosen ...
- 3,682
9
votes
1
answer
3k
views
Variance-covariance matrix for ridge regression with stochastic $\lambda$
In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
- 4,377
9
votes
0
answers
754
views
The Regularization Path for Smoothing Splines
I've got a potentially interesting question. Does anyone know if R already has a package for calculating the entire regularization path of the smoothing spline?
That is, for:
$$\hat{f}_{\lambda}=...
- 1,254
8
votes
3
answers
4k
views
In Ridge regression and LASSO, why smaller $\beta$ would be better?
Can anyone provide an intuitive view on why it is better to have smaller beta?
For LASSO I can understand that, there is a feature selection component here. Less features make the model simpler and ...
- 1,489
8
votes
5
answers
8k
views
Mean squared error of OLS smaller than Ridge?
I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
- 135
8
votes
4
answers
566
views
Ridge regression: regularizing towards a value
The traditional ridge regression estimate is
$$
\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1} X^T Y
$$
which comes from adding the penalty term $\lambda ||\beta||^2_2$.
I have been struggling to ...
- 113
8
votes
1
answer
5k
views
Optimizing the ridge regression loss function with unpenalized intercept
I've been trying to optimize the ridge regression loss function by gradient descent, but I've been banging my head against it for a while. The loss function can be written as
$$L(w,b)=\sum_{i=1}^n(y^{...
- 183
8
votes
1
answer
6k
views
Deep Learning: Use L2 and Dropout Regularization Simultaneously?
Is there a theoretical basis against using both L2 and Dropout regularization simultaneously for training a deep neural network? They are both related but could they be complementary if used together?...
- 455
8
votes
2
answers
5k
views
Lasso and Ridge tuning parameter scope
In ridge and lasso linear regression, an important step is to choose the tuning parameter lambda, often I use grid search on log scale from -6->4, it works well on ridge, but on lasso, should I take ...
- 233
8
votes
1
answer
4k
views
AIC and its degrees of freedom for linear regression models
I have a dataset $S$ with $D$ features and three fitted linear regression models:
Model1. Ridge regression that is fitted on all $D$ features from $S$.
Model2. Ridge regression that is fitted on some $...
- 988
8
votes
1
answer
9k
views
comparing OLS, ridge and lasso
I am trying to compare OLR, ridge and lasso in my situation. I could calculate SE for OLR and lasso but not for ridge. The following is Prostrate data from ...
- 2,436
8
votes
1
answer
10k
views
What does it mean to have a "gaussian prior?"
When reading up on ridge regression, I saw it stated that it has a "gaussian prior." I realized that I don't know what the word prior means in this context and what it is applied to?
I ...
- 434
8
votes
2
answers
13k
views
Can ridge regression be used in the presence of categorical predictors?
I have a regression problem and I am thinking of using ridge regression. One of the predictors is subject's gender, which is a categorical variable. How to take care of this variable for ridge ...
- 4,067
8
votes
1
answer
2k
views
Grid fineness and overfitting when tuning $\lambda$ in LASSO, ridge, elastic net
I wonder about
the optimal grid fineness and
what the relation between grid fineness and overfitting is
in regularization methods such as LASSO, ridge regression or elastic net.
Suppose I want ...
- 67.5k
8
votes
2
answers
5k
views
Confused by MATLAB's implementation of ridge
I have two different implementations of ridge in MATLAB. One is simply
$\mathbf x = (\mathbf{A}'\mathbf{A}+\mathbf{I}\lambda)^{-1}\mathbf{A}'\mathbf b$
(as seen ...
- 81
8
votes
1
answer
2k
views
Equivalence between Elastic Net formulations
According to Hastie's paper, the elastic net has two equivalent formulations:
$$\hat{\beta} = \underset{\beta}{\operatorname{argmin}} \left\{ \sum_{i=1}^N\left(y_i-\sum_{j=1}^p x_{ij} \beta_j\right)^...
- 344
8
votes
1
answer
7k
views
Support vector regression versus kernel ridge regression
I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully).
Let's begin with ...
- 259
8
votes
1
answer
414
views
Scalable multinomial regression implementation
I need to do a high dimensional biological data analysis. My data consists of hundreds of thousands of dimensions. I am looking for an implementation of multinomial logistic regression that will scale ...
- 1,683
7
votes
2
answers
1k
views
Lasso coefficient for some features is higher than Linear Regression Coefficient
I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, ...
- 173
7
votes
4
answers
22k
views
How to interpret ridge regression plot
Following is the ridge regression example in MASS package:
...
- 10k
7
votes
3
answers
27k
views
Ridge Regression -Increase in $\lambda$ leads to a decrease in flexibilty
In Introduction to Statistical Learning, in the part where ridge regression is explained, the authors say that
As $\lambda$ increases, the flexibility of the ridge regression fit
decreases, ...
- 787
7
votes
3
answers
3k
views
Bayesion priors in ridge regression with scikit learn's linear model
I'm using scikit learn's linear model to do ridge regression.
Ridge regression penalizes parameters for moving away from zero. I want to penalize for moving away from a certain prior, with each ...
- 81
7
votes
1
answer
16k
views
Sparsity in Lasso and advantage over ridge (Statistical Learning) [duplicate]
I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is ...
- 9,157
7
votes
1
answer
885
views
Why does sklearn Ridge not accept warm start?
I am experimenting with some regularized linear regression methods using sklearn and noticed that Ridge does not accept warm start. I found it odd as many other methods do accept like Lasso, ...
- 115
7
votes
2
answers
2k
views
Is there a mathematical expression that shows how LASSO shrinks coefficients (including some to zero)?
By using singular value decomposition (SVD), I noticed from the derivation that ridge regression shrinks the coefficients by factor $\frac{D^2}{D^2+\lambda}$, where $D$ is the diagonal matrix of the ...
- 147
7
votes
2
answers
437
views
Is Cross Validation useless unless the Hypotheses are nested?
If I generate many random models (without considering the data at all) in a regression setting simply by randomly assigning coefficient values and then evaluating these models over the dataset with an ...
- 4,486
7
votes
1
answer
879
views
Effective degrees of freedom for regularized regression
If I have a quadratic programming problem
$$\min_b \frac{1}{2} b^tX^tXb - b^t(X^tY)$$
which I regularize by adding a multiple of the identity$\lambda I$ to $X^tX$, then the effective degrees of ...
- 3,682
7
votes
3
answers
3k
views
Why does regularization of coefficient magnitude improve the generalization of linear regression? [duplicate]
What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that
In many cases, tikhonov matrix is chosen as ...
- 502
7
votes
1
answer
409
views
How to decide which penalty measure to use ? any general guidelines or thumb rules out of textbook
A number of regularization measures are available in literatures, which is kind of confusing to beginners. The classical penalty is ridge by Hoerl & Kennard (1970,Technometrics 12, 55–67).
...
- 2,258
7
votes
2
answers
562
views
Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?
I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
- 719
7
votes
1
answer
2k
views
Computing cross-validated $R^2$ from mean cross-validation error
I am currently using cv.glmnet in R. I would like to compute both a training $R^2$ and a cross-validated $R^2$. R gives mean cross-validated error and for the ...
- 1,485
7
votes
1
answer
2k
views
Ridge Regression and Lasso Regression
I am currently working on this problem and the goal is to develop a linear regression model to predict my Y(blood pressure) with 8 predictors, using Ridge & Lasso regression. I begin by examining ...
- 83
7
votes
1
answer
2k
views
Bias and variance properties of $L^1$ vs $L^2$ normalization
When moving from $L^2$ to $L^1$ normalization in Linear Regression, should I expect to see more bias or more variance? Note that bias is a sign of under fitting and variance is a sign of over fitting. ...
- 71
7
votes
1
answer
2k
views
Estimating the prediction variance in kernel ridge regression
I'm trying to estimate the variance of predictions for a kernel ridge regression model.
The model is simply kernel ridge regression:
$$\hat{y} = K(K+\lambda I)^{-1}y = A y$$
$K$ is the $n \times n$ ...
- 1,042
7
votes
1
answer
863
views
Ridge regression in multivariate Gaussian distribution
When implementing GMM (Gaussian Mixture Model) in practice, the covariance matrix ${\Sigma}_{D\times D}$ is often singular. The reason is that we have to estimate $\frac{D(D+1)}{2}$ parameters in $\...
- 171
7
votes
1
answer
348
views
Selecting optimal set of eigenvectors for Principal Components Regression
I am testing various techniques for dealing with strong multi-collinearity (MC) in a regression problem.
There have been various comparison papers written between competing techniques such as Ridge ...
- 1,763
7
votes
1
answer
6k
views
How do you interpret the results from ridge regression?
I started learning ridge regression in R. I applied the linear ridge regression to my full data set and got the following results.
...
- 81
7
votes
0
answers
321
views
Nonnegative identity-link Poisson regression with ridge or fused ridge penalty
I would like to fit nonnegative identity-link Poisson regression models with a ridge or fused ridge penalty, i.e. with nonnegativity constraints on the fitted coefficients, Poisson error noise & a ...
- 3,210
7
votes
0
answers
1k
views
Geometrical interpretation of L1 regression
I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
- 1,304
7
votes
0
answers
290
views
Reference Request: Information Geometry for Ridge Regression
I am reading the book "regression estimators" by Gruber 2010 where he uses this technique to compare Ridge Regressors, however he concentrates on deriving the mathematical results without ...
- 1,763
6
votes
3
answers
2k
views
Can I use lasso when it is not a high dimensional setting?
I have 500 observations and 200 predictors, and I want to do the prediction while selecting some important features. I know that regularisation method (ridge, lasso) are shrinkage method for high-...
- 147
6
votes
3
answers
3k
views
Is there any special case where ridge regression can shrink coefficients to zero?
Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ?
It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge ...
- 231
6
votes
1
answer
36k
views
Alpha parameter in ridge regression is high
I am using the Ridge linear regression from sickit learn. In the documentation they stated that the alpha parameter has to be small.
However I am getting my best model performance at 6060. Am I doing ...
- 233
6
votes
2
answers
1k
views
Bayesian interpretation of logistic ridge regression
Most textbooks (also this blog) cover the fact that ridge regression,
$$
\hat y = \hat \beta X;
\\
\hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2}
+ \lambda \...
- 8,997
6
votes
1
answer
524
views
Ridge or multiple linear regression following PCA?
I have a real world clinical dataset with a severe issue of p >> n. I have thus decided to run PCA before modelling the data. This leads to a dataset with 150 samples with 85 features.
I would ...
- 335
6
votes
1
answer
4k
views
Is this the correct way to run an adaptive LASSO?
I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
- 2,621
6
votes
1
answer
628
views
When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?
Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
- 177
6
votes
4
answers
2k
views
Rationale behind shrinking regression coefficients in Ridge or LASSO regression
I understand that with Ridge or Lasso regression we are trying to shrink regression coefficients, and we specify the amount of shrinking we need by varying alpha. But I cannot understand the intuition ...
- 83