Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
625
questions
0
votes
0answers
12 views
Implementing L1 and L2 penalty in sequential coordinate descent least squares
I'm trying to implement L1 and L2 regularization in a fast RcppArmadillo function for non-negative least squares.
The function below is adapted from the NNLM R package, receives an initial value for <...
0
votes
0answers
16 views
Why wouldn't you perform PCA before performing ridge regression on highly correlated parameters?
I'm trying to wrap my head around the L2 regularization component in ridge regression, to build a model on noisy, correlated data.
I understand the $\lambda$ introduces a penalty for high bias during ...
0
votes
0answers
8 views
Standardized inputs in KRR
Say we have N observations of a function f and that one wishes to obtain an approximation of f by using Kernel Ridge Regression. I read that it was recommended to standardize the inputs. So, when one ...
1
vote
1answer
15 views
Numerical solution to the constrained ridge regression
The constrained ridge regression problem is of the form:
$\arg\min_{\|\beta\|_2\le t}\|X\beta-y\|_2$. Given a matrix $X$, a vector $y$ and the constrain parameter $t$, how do you solve it numerically?
...
0
votes
1answer
36 views
Bias of Ridge Estimates in Regression
For a given ridge parameter, ridge estimates minimize the sum of squared predictions subject to an inequality constraint.
Are the ridge estimates biased regardless of whether the aforementioned ...
6
votes
1answer
847 views
Why regularization parameter called as lambda in theory and alpha in python?
I was learning about regularization and came across the term called regularization parameter.
I see that it is called lambda in ...
0
votes
1answer
45 views
Why feature selection using `L1` and not using `L2` norm? [duplicate]
I read a tutorial here. In which, I came across the below plots
I read an explanation quoted ...
1
vote
0answers
43 views
L1 vs L2 norm - Circle and Diamond [duplicate]
I am new to ML and recently came across the L1 and L2 norm.
The tutorials that I read here and here show some circle and diamond ...
5
votes
0answers
102 views
Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?
I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I ...
1
vote
1answer
25 views
In which cases should I split the data in training set and test set [closed]
I am taking a course on machine learning and in one problem I should perform a Ridge regression to fit some given data to a known model. I was wondering if, in this case, there are any advantage in ...
3
votes
1answer
39 views
On a Ridge regression-like problem
I am trying to implement a sort of Ridge regression for the following problem
\begin{equation}
y = a^\top X b,
\end{equation}
where $y\in \mathbb{R}$, $a\in\mathbb{R}^M$, $b\in\mathbb{R}^N$ and $X\in\...
0
votes
0answers
22 views
Extreme learning machine with ridge regression
I'm new to extreme learning machine (ELM) which is a single layer feedforward neural network. I'm trying to write a ridge version of the classical ELM. But this Ridge-ELM confuses me. I think the ...
1
vote
3answers
63 views
How to select the best feature set from Ridge regression?
I have applied L2 regularization on my features and have got coefficient values as below(hiding the column name due to client work:
I am unsure about what all features should I choose? Should I ...
0
votes
0answers
14 views
Interpreting coefficient values in lasso or ridge regression
I am doing feature selection on a dataset using Lasso and Ridge regression. I have already standardized the Features, except the Target feature. I am new to this, so just wanted to check that after ...
2
votes
0answers
18 views
Intuition behind the degrees of freedom in ridge regression
I'm reading through the ESL book and I'm on the part of ridge regression where the effective degrees of freedom are defined
$$
df(\lambda) = tr(X(X'X + \lambda I)^{-1}X') = \sum_{j=1}^p{\frac{d_j^2}{...
0
votes
1answer
21 views
Ridge Regression - Advice on Modeling Sales Data
I am looking to use ridge regression to predict end of quarter sales revenue. My features are sales pipeline and revenue booked quarter to date. As the quarter progresses sales pipeline will ...
1
vote
0answers
41 views
Why Weighted Ridge Regression gives same results as weighted least squares only when solved iteratively?
I was experimenting with weighted ridge regression for a linear system, where the closed-form solution is given by:
$$
b =(X^T WX + \lambda I)^{-1}X^T W y
$$
and also weighted least squares whose ...
0
votes
1answer
31 views
Can we use Gradient Descent in the place of Ridge Regression in overfitting problem while doing linear regression problem?
What is the difference between Gradient Descent and Ridge regression?
We use ridge regression for overfitting problem when the Mean Squared Error for test dataset is high. I think that we can use ...
0
votes
0answers
6 views
Given two equations that differ by one predictor, under ridge regression, which estimates are generally larger in magnitude?
Suppose we have two equations
$$
Y=\alpha_1X_1+\alpha_3X_3
$$
and
$$
Y=\beta_1X_1+\beta_2X_2+\beta_3X_3
$$
Suppose further that $X_1=X_2$, then would it usually be the case that $\hat{\alpha_1}$ or $\...
1
vote
1answer
40 views
norm of ridge regression estimator
is there a characterization or an upper bound on the norm of the ridge regression estimator (coefficients)? As the Tikhonov regularization attempts to regularize these coefficients as part of the ...
0
votes
0answers
11 views
Choosing the regularisation parameter - L-curve for multiple models
The problem appears to be quite simple, and well-studied, but I'm stuck. I minimise the misfit function in LS sense, and I add regularisation term. This, however, has an objective to keep the misfit ...
0
votes
0answers
15 views
Would you please give some examples on Bias-Variance Trade off? [duplicate]
I am a new learner for Machine Learning and are confused about the idea of bias-variance trade-off. Would you please offer some specific examples or situations that a bias-variance trade-off occurs?
...
2
votes
1answer
70 views
Properties of ridge regression hat matrix and ridge residuals
I'm referencing https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 34 question 1.5 we need to prove :
Ridge fit $\widehat{Y}(\lambda)=X(X^{\top}X+\lambda I_p)^{-1}X^{\top}Y$ is not ...
1
vote
0answers
75 views
How to make the regressor of LASSO consistent?
Suppose there is one regressor $X$ ith a sample so that $\sum_{i=1}^n X_i^2=n$. And suppose the OLS estimator of $Y$ on $X$ (no intercept) is consistent. What condition does $\lambda$ need to satisfy ...
0
votes
0answers
21 views
RSS of ridge regression in terms of OLS estimator
In the work by Hoerl, Arthur E., and Robert W. Kennard , "Ridge regression: Biased estimation for nonorthogonal problems." the following formula (3.1) is presented:
$$ RSS=(Y-XB)'(Y-XB)=(Y-X\...
0
votes
0answers
13 views
Why does the value of the penalized ridge is divided by 2 in GLMNET? [duplicate]
If you look at GLMNET Vignette, it shows that it solves for the gaussian case:
But why does it divide the value of $\parallel \beta \parallel_2^2$ by 2?
1
vote
0answers
31 views
Relationship between Bias/Variance and Covariates in Ridge/Lasso Regression
Suppose I add irrelevant (i.e. no explanatory power) regressors to a ridge/lasso regression. Does this impact the model bias/variance?
In the case of OLS, the model bias remains unchanged, while the ...
1
vote
1answer
33 views
Intersection of MSE-Loss and Regularization Term
In several questions [1,2] the graphical intuition of the L1/L2 regularization has been discussed. But, for example in [1], it has been stated that:
The solution to the constrained optimization lies ...
1
vote
2answers
61 views
Lasso Regression's role in shrinking the coefficient to zero and Ridge Regression's in not doing the same
How Lasso regression helps feature selection of model by making the coefficient zero?
I could see few below with below diagram. Can anyone explain in simple terms how to correlate below diagram with i....
3
votes
2answers
77 views
How to prove ridge estimator residuals variance
The ridge residuals are defined as $\epsilon(\lambda)=y-X\beta^{ridge}(\lambda)$, for the model $y_i=x_i^T\beta+e_i$, where $e_i\sim N(0,\sigma^2)$, and $\beta$ is estimated by the ridge regression ...
11
votes
2answers
843 views
Proving Ridge Regression is strictly convex
Definition of ridge regression
$$min_\beta||y-X\beta||_2^2+\lambda||\beta||_2^2, \lambda\ge0$$
you can prove a function is strictly convex if the 2nd derivative is strictly greater than 0 thus
But ...
2
votes
1answer
83 views
Ridge fit is not orthogonal to ridge residuals
So I'm reading https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 8 under Example 1.3 it says
From the figure it is obvious that for any $\lambda >0$ the ‘ridge fit’ $\widehat{Y}(\...
2
votes
1answer
27 views
Logistic regression - regularized model always predict lower probabilities on average compared to unregularized model
I have a model that is using L2 regularization. The non-regularized model has a few coefficients with a high positive value, but otherwise the features have very similar coefficients. In the ...
0
votes
0answers
23 views
MSE as a function of the penalty: How to deal with multiple Minima?
The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7.
These are made-up data I wanted to use in an ...
1
vote
0answers
22 views
Ridge Regression in R: MASS implementation vs User Defined returning different coefficients
I am currently having trouble with the results of a user defined ridge regression function (one that I have created) against that of the MASS::lm.ridge() function. Below is what is defined as my data:
...
1
vote
1answer
111 views
Regularised linear regression with Newton's method?
I am trying to use the Newton's method
$\theta^{(t+1)} = \theta^{(t)} - [H^{(t)}]^{-1} [\nabla L(\theta^{(t)})]$ to minimise the following loss fucntion
$L(\theta) = (y - X\theta)^T(y-X\theta) + \...
0
votes
0answers
12 views
Which model to use? (cross validation with early stopping)
In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
2
votes
0answers
44 views
What is the intuition of a dual?
I have been hearing that the Ridge regression is the dual to the GP (Gaussian process regression). What does this mean? Can someone please give an intuition on what 'dual' is.
My impression of the '...
0
votes
1answer
129 views
Ridge and Quadratic Programming for Portfolio Norm Optimization
Much like this post: Quadratic Programming and Lasso, I'm trying to integrate RIDGE Penalty in a dedicated quadratic solver. In my case, I am working with quadprog from MATLAB. Unlike LASSO where you ...
2
votes
1answer
66 views
Ridge regression not working for very simple dataset (yet exact same code works on another dataset)
I found some R code for performing ridge regression on the BostonHousing dataset. I tried to use the exact same code on some simple artifical data but it fails and I get the error message: ...
0
votes
0answers
22 views
Equal variance along left and right singular vectors?
Please confirm or reject my line of reasoning:
Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$,
Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, ...
3
votes
1answer
84 views
Eigenvalues in Ridge regression [duplicate]
The ridge regression estimate is given by
$$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$
where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
0
votes
1answer
41 views
Multiplying a predictor by a constant in Lasso/Ridge regression
If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
3
votes
1answer
51 views
Ridge regularization - intuition behind $\lambda$
I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
3
votes
1answer
49 views
Details about Ridge regression [duplicate]
I have a question about the mathematical details of Ridge Regression and I have not been able to find a detailed explanation. For what I know the ridge regression is a penalty term that is used to ...
0
votes
0answers
22 views
Normalize parameter in sklearn Ridge, Lasso, ElasticNet [duplicate]
Is there any risk or disadvantage to set normalize=True when using ridge, lasso or elasticnet or does it only have benefits? And what is the impact on the range of alpha if it is set to True, does it ...
0
votes
0answers
46 views
Box-Cox formation with model selection, regularization, etc
As my data is not normally distributed, I performed the Box-Cox Transformation on the response.
...
0
votes
0answers
29 views
Which one (ridge or lasso) focuses more on the weights that higher?
From my understanding lasso more aggressive bring the weights to zero when the weights are less than 1. While, ridge will more aggressive bring weights to 0 (I know that ridge won't actually bring ...
1
vote
0answers
28 views
Kernel ridge regression and Gaussian Process Regression
One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same.
It may be a very naive question but why?
To me, they are quite ...
6
votes
5answers
837 views
Mean squared error of OLS smaller than Ridge?
I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
