# Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

602 questions
Filter by
Sorted by
Tagged with
10 views

### How to derive the closed-form expression for a ridge regression estimator with an extra vector in the penalty term?

where $\beta_0$ is a known fixed vector in $R^p$. I understand the derivation for the closed-form expression of the ridge regression estimator: But I cannot find anything about when the penalty term ...
26 views

10 views

### Which model to use? (cross validation with early stopping)

In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
40 views

### What is the intuition of a dual?

I have been hearing that the Ridge regression is the dual to the GP (Gaussian process regression). What does this mean? Can someone please give an intuition on what 'dual' is. My impression of the '...
73 views

### Ridge and Quadratic Programming for Portfolio Norm Optimization

Much like this post: Quadratic Programming and Lasso, I'm trying to integrate RIDGE Penalty in a dedicated quadratic solver. In my case, I am working with quadprog from MATLAB. Unlike LASSO where you ...
48 views

### Ridge regression not working for very simple dataset (yet exact same code works on another dataset)

I found some R code for performing ridge regression on the BostonHousing dataset. I tried to use the exact same code on some simple artifical data but it fails and I get the error message: ...
13 views

### Equal variance along left and right singular vectors?

Please confirm or reject my line of reasoning: Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$, Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, ...
45 views

### Eigenvalues in Ridge regression [duplicate]

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
21 views

### Multiplying a predictor by a constant in Lasso/Ridge regression

If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
40 views

### Ridge regularization - intuition behind $\lambda$

I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
42 views

### Details about Ridge regression [duplicate]

I have a question about the mathematical details of Ridge Regression and I have not been able to find a detailed explanation. For what I know the ridge regression is a penalty term that is used to ...
14 views

### Normalize parameter in sklearn Ridge, Lasso, ElasticNet [duplicate]

Is there any risk or disadvantage to set normalize=True when using ridge, lasso or elasticnet or does it only have benefits? And what is the impact on the range of alpha if it is set to True, does it ...
30 views

### Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
17 views

### Which one (ridge or lasso) focuses more on the weights that higher?

From my understanding lasso more aggressive bring the weights to zero when the weights are less than 1. While, ridge will more aggressive bring weights to 0 (I know that ridge won't actually bring ...
26 views

### Kernel ridge regression and Gaussian Process Regression

One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same. It may be a very naive question but why? To me, they are quite ...
304 views

### Mean squared error of OLS smaller than Ridge?

I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
90 views

### Is this the correct way to run an adaptive LASSO?

I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
33 views

### Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution. The question is For some dataset $X$ with response $Y$, you apply ridge regression. ...
18 views

304 views

### How to implement the closed form solution of Ridge Regression in Python when intercept is not 0 (fit_intercept=True) without using sklearn?

The well-known closed-form solution of Ridge regression is: I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no ...
29 views

### Mixed-effects models with lasso penalty and ridge penalty in R [closed]

I am using the PISA 2015 data and trying to run a mixed-effects ridge and lasso regression model Schools will be included as a random effect, and student-level (e.g. motivation, socioeconomic status, ...
37 views

### Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?

I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
30 views

### standardizing the MSE

I conducted a Ridge regression with k-fold validation. All the predictors were scaled prior to the regression procedure. To report on the accuracy of my model's prediction, I calculated the MSE in the ...
23 views

### Misclassification for test and training sets

I have a problem where I need to provide the misclassification error for both training- and test-set. I am working with logistic regression, so I have a binomial family for my models. I have two ...
28 views

### Why Ridge regression doesn't depend on centering $y$ in sklearn?

In sklearn's manual for Ridge they wrote the following about its parameter "fit_intercept": But it seems that Ridge model doesn't depend on whether $y$ is centered or not: ...
33 views

### Bias and variance calculation for Ridge estimator of β

I understand how bias and variance for ridge estimator of β are calculated when the model is Y=Xβ + ϵ. But I have the model Y=Xtβ + ϵ. I don't understand if a model like that makes sense, can someone ...
14 views

### Ridge regression on one predictor and the reduction of the risk of overfitting

In the book Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 1, the author stated that when doing Ridge Regression (for only one predictor), this regularization reduces the risk of ...
### In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random?
I am having a hard time understanding the Bayesian Ridge Regression. In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random? or do we treat the $\lambda$ as given?...