A regularization method for regression models that shrinks coefficients towards zero.

learn more… | top users | synonyms

1
vote
1answer
22 views

What happens if you square an RBF kernel function?

Let's say we use a kernel regularization algorithm such as ridge regression to minimize some loss in an RBF kernel: $$\min_{h \in H} \frac{1}{n} \sum_i (h(x_i) - y(x_i))^2 + ||h||^2_K$$ We get some ...
0
votes
0answers
13 views

Cross-validation results differ from package

I am trying to test out a hand-rolled cross-validation procedure for ridge regression. I've run the glmnet package which gives me an MSE bottoming out around ...
1
vote
1answer
49 views

Deriving the Ridge Regression $\boldsymbol{\beta}\mid \mathbf{y}$ distribution

Apparently the estimate $\hat{\boldsymbol{\beta}}$ for ridge regression comes up as the mean or mode of the posterior distribution given by $f_{\boldsymbol{\beta}\mid \mathbf{y}}$. This is the ...
1
vote
3answers
73 views

Why are solution to ridge regression always expressed using matrix notation?

Consider the following ridge regression problem: minimize the loss function $\sum_{i=1}^n ||y_i - w^T x_i||_2^2 + \lambda ||w||_2^2$ with respect to the weight vector w. Taking derivative with respect ...
3
votes
1answer
39 views

Ridge regression formulation as constrained versus penalized: How are they equivalent?

I seem to be misunderstanding a claim about linear regression methods that I've seen in various places. The parameters of the problem are: Input: $N$ data samples of $p+1$ quantities each ...
3
votes
2answers
103 views

Ridge regression — why does the model only care to control large outliers?

One of the purposes of ridge regression is to curb the effects of outliers which may cause the regression coefficients to be so large and hence cause a highly biased model. That's why the constraint ...
7
votes
3answers
215 views

How to find regression coefficients $\beta$ in ridge regression?

In ridge regression, the objective function to be minimized is: $$\text{RSS}+\lambda \sum\beta_j^2.$$ Can this be optimized using the Lagrange multiplier method? Or is it straight differentiation?
1
vote
0answers
52 views

Alternatives to Box-Tidwell transformation for ridge regression?

I would like to fit the following model by ridge regression (the xs correlate strongly with one another) $y = \beta_1 {x_1}^{\lambda_1} + \beta_2 {x_2}^{\lambda_2} + \beta_3 {x_3}^{\lambda_3} + ...
0
votes
0answers
21 views

Why is intercept excluded in ridge regression? [duplicate]

Why don't we include the intercept term in ridge regression, i.e exclude $w_0$ term in calculating the coefficients? Why is it so?
0
votes
0answers
24 views

The scaling problem of ridge regression

I have been confused with the scaling of ridge regression input for a long time. There are several sources about how to do the scaling: Just do the centering to input(From "The Elements of ...
2
votes
1answer
48 views

Decompose ridge regression bias error into model bias and estimation bias

How can I show that the in-sample bias error in Ridge regression can be decomposed into model bias plus estimation bias? I.e., if $Avg$ takes the average over all the input variables $x$ in the ...
6
votes
1answer
130 views

Computing cross-validated $R^2$ from mean cross-validation error

I am currently using cv.glmnet in R. I would like to compute both a training $R^2$ and a cross-validated $R^2$. R gives mean cross-validated error and for the ...
1
vote
0answers
35 views

Meaning and significance testing of coefficients in lasso/ridge regression

Can somebody explain the importance of significance testing in ridge/lasso regression? Is it necessary to do it? And, how can we interpret the coefficients of ridge regression which are penalized?
2
votes
1answer
66 views

Elastic net produces complex output with too many non-zero coefficients

I have run 3-fold cross-validation for elastic net using elasticnet R function on ~200 observations and using 80 variables (and there will be some more). Both ...
3
votes
1answer
223 views

Why is ridge regression giving different results in Matlab and Python?

Why is the output from Matlab and Python vary for ridge regression? I use the ridge command in Matlab and scikit-learn in Python ...
3
votes
2answers
61 views

Can ridge regression be used in the presence of categorical predictors?

I have a regression problem and I am thinking of using ridge regression. One of the predictors is subject's gender, which is a categorical variable. How to take care of this variable for ridge ...
1
vote
0answers
46 views

First step of Kernel Ridge Regression?

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
0
votes
0answers
10 views

First step of Kernal Ridge Regression? [duplicate]

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
0
votes
1answer
62 views

Do methods exist other than Ridge Regression and Y ~ X + 0 to prevent OLS from dropping variables?

Goal is to evaluate chess players using a novel analysis system I'm been working on -- not all wins are created equal, finding the only move in razor sharp positions is better than finding the best ...
7
votes
2answers
382 views

What is elastic net regularization, and how does it solve the drawbacks of Ridge (L2) and Lasso (L1)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
9
votes
2answers
511 views

When will L1 regularization work better than L2 and vice versa?

Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant. How to decide which regularization (L1 or L2) to ...
2
votes
0answers
10 views

How to model the discrepancy between test result and prediction code, and find the main sources of discrepancy

I have data about some tested machines, stored as a database with $\approx$ 300 observations, 40 predictors, 2 outcomes. The two outcomes (responses) are the result of the machine test for that ...
0
votes
0answers
15 views

Ridge regression on centred or uncentered data? [duplicate]

Today is the first day that I have heard about ridge regression, so please do not judge harshly. I have learned that you use ridge regression when there is a case for multicollinearity of your ...
2
votes
1answer
126 views

Why is using centered or uncentered data equivalent in ridge regression?

Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: \begin{equation} (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} ...
1
vote
0answers
123 views

Implementing kernel ridge regression

I want to implement kernel ridge regression in R. My problem is that I can't figure out how to generate the kernel values and I do not know how to use them for the ridge regression. Before going to ...
2
votes
1answer
93 views

Recommend a method for variable selection (other than classification tree or random forest)?

Just wonder if you could recommend a few methods (other than tree-based methods) to analyze a dataset in which n= 350 and p = 35. The goal is not so much about prediction, but to find/select ...
25
votes
2answers
1k views

Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
4
votes
1answer
179 views

Why can't ridge regression provide better interpretability than LASSO?

I already have an idea about pros and cons of ridge regression and the LASSO. For the LASSO, L1 penalty term will yield a sparse coefficient vector, which can be viewed as a feature selection method. ...
5
votes
2answers
327 views

Naive Ridge Regression in R?

I'm trying to learn some basic Machine Learning and some basic R. I have made a very naive implementation of $L_2$ regularization in R based on the formula: $\hat w^{ridge} = (X^TX +\lambda I)^{-1} ...
12
votes
1answer
216 views

When is nested cross-validation really needed and can make a practical difference?

When using cross-validation to do model selection (such as e.g. hyperparameter tuning) and to assess the performance of the best model, one should use nested cross-validation. The outer loop is to ...
0
votes
1answer
46 views

Ordinary linear regression vs penalized regression: predictive MSE

Does predicting on a test set with an ordinary linear regression model result in a smaller predictive MSE compared to a penalized regression model (LASSO or ridge)?
1
vote
1answer
71 views

Estimate the tuning parameter in Ridge logistic regression

I need to know how can we estimate the tuning parameter in penalized likelihood? I write my own code but there is a mistake I could not find it.!! Could you please help, I do my best to make the code ...
0
votes
0answers
35 views

Ridge Regression, multi class classification

I am a beginner in the field of data mining. I have started of with regression concepts. From my understanding we use linear regression for prediction and logistic for classification. I've gone ...
4
votes
1answer
333 views

Why will ridge regression not shrink some coefficients to zero like lasso?

When explaining LASSO regression, the diagram of a diamond and circle is often used. It is said that because the shape of the constraint in LASSO is a diamond, the least squares solution obtained ...
2
votes
1answer
332 views

cv.glmnet Ridge Regression lambda.min = lambda.1se?

I'm currently running a ridge regression in R using the glmnet package, however, I recently ran into a new problem and was hoping for some help in interpreting my ...
0
votes
1answer
38 views

Ridge regression via OLS with “phoney data”

*********Okay so I figured out what was wrong! I wasn't centering the date like the lm.ridge function does. However I still cannot reproduce the intercept that lm.ridge gives me. According to my ...
1
vote
1answer
282 views

Ridge regression in R with p values and goodness of fit

Doing ridge regression in R I have discovered linearRidge in the ridge package - which fits a model, reports coefficients and ...
0
votes
0answers
50 views

R - Manually Adding Intercept to glmnet Ridge Regression

Is there a way to manually add an intercept term manually into the glmnet function of R rather than using the built-in intercept ...
13
votes
1answer
296 views

What are the assumptions of ridge regression and how to test them?

Consider the standard model for multiple regression $$Y=X\beta+\varepsilon$$ where $\varepsilon \sim \mathcal N(0, \sigma^2I_n)$, so normality, homoscedasticity and uncorrelatedness of errors all ...
0
votes
1answer
97 views

Linear Ridge not correct prediction/coefficients- Scikit learn

I am using a similar code to this ridge example. The code proposed is simple. X and Y points inside [-1,1] range and predict the radius creating polynomial features and ridge linear regression. As ...
0
votes
1answer
122 views

Alpha parameter in ridge regression is high

I am using the Ridge linear regression from sickit learn. In the documentation they stated that the alpha parameter has to be small. However I am getting my best model performance at 6060. Am I doing ...
1
vote
1answer
57 views

Should the lambda of ridge regression be related to number of data points?

Suppose we have data $ (x_{1}, y_{1})\ldots (x_{N}, y_{N})$. The loss function of ridge regression is $$ \sum_i^N{(y_i - x^T_i\mathbf{\beta})^2} + \lambda \sum_j^p{\beta^2_j} $$ Notice that $ ...
0
votes
0answers
21 views

Strange selection results, ridge logistic regression

I'm studying ridge logistic regression with glmnet on R. I have a lot of regressors which are dummies. I'm trying to maximise the AUC (prediction of a binary output). My question is about the graph ...
0
votes
1answer
32 views

MATLAB - using ridge regression weights

One simple and straightforward question, which is confusing me because poor results that I'm getting. I'm using MATLAB's built in ridge function to get weights for my model, on my training dataset. ...
1
vote
2answers
183 views

Can someone explain what the foldid argument in glmnet does?

I m trying to determine what alpha to use in my glmnet function, but the help file tells me: Note that cv.glmnet does NOT search for values for alpha. A ...
2
votes
1answer
87 views

Estimating the prediction variance in kernel ridge regression

I'm trying to estimate the variance of predictions for a kernel ridge regression model. The model is simply kernel ridge regression: $$\hat{y} = K(K+\lambda I)^{-1}y = A y$$ $K$ is the $n \times n$ ...
1
vote
1answer
83 views

How to calculate Hat matrix for penalized spline regressions?

The book "Semiparametric Regression" by Ruppert et al. (2003) provided a computationally fast algorithm for Penalized Spline Regression. I put a part of the algorithm here. Does anybody can do algebra ...
20
votes
2answers
239 views

confidence intervals' coverage with regularized estimates

Suppose I'm trying to estimate a large number of parameters from some high-dimensional data, using some kind of regularized estimates. The regularizer introduces some bias into the estimates, but it ...
2
votes
0answers
53 views

Is there a clear set of conditions under which lasso, ridge, or elastic net solution paths are monotone?

The question What to conclude from this lasso plot (glmnet) demonstrates solution paths for the lasso estimator that are not monotonic. That is, some of the cofficients grow in absolute value before ...