Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

When to use ridge regression or lasso rather than elastic net?

If one has no ex-ante information about what the L1-ratio hyperparameter should be in the context of elastic net regularization, when should one instead use lasso or ridge? This ratio is referred to ...
2
votes
0answers
39 views

Can we estimate independent parameters when $p > n$?

I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals. I know that when we estimate fixed effects, the number of ...
0
votes
2answers
97 views

Orthogonality of columns of the augmented design matrix for ridge regression

In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
1
vote
1answer
34 views

Ridge and Lasso in GLMs

In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others ...
0
votes
0answers
28 views

Inner Product RKHS and regression

Say you want to solve the kernel ridge regression as follows: $\min\limits_{f\in H_K} \left[\lambda||f||^{2}_{H_K} + \sum\limits_{i=1}^{N}(y_i - f(x_i))^2\right]$, where $\lambda > 0$. We know how ...
1
vote
1answer
59 views

Why does ridge regression only have one hyperparameter $\lambda$?

Ridge Regression objective$$\underset{\beta}{\text{min}} \sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$ SVM primal problem: $$\begin{align} \max_{\mathbf{\alpha}} \quad &\min_{\...
5
votes
1answer
103 views

When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?

Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
2
votes
1answer
80 views

Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?

I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
0
votes
0answers
47 views

Why does area under curve not change from 0.5?

I have performed a ridge logistic regression with glmnet and now I look at the performance metric AUC. The script is: ...
1
vote
1answer
35 views

How to optimise penalty parameter in ridge regression using AIC

So I know for a ridge regression model, we need to find an optimal $\lambda$ value. I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
1
vote
1answer
28 views

LassoCV vs RidgeCV in Python — why are their default number of folds different?

In https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html, it says that LassoCV defaults to 5 folds. In https://scikit-learn.org/stable/...
1
vote
0answers
12 views

How do the training and cross validation mean squared error curves behave as a function of $\lambda$?

I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
2
votes
2answers
72 views

Bias variance tradeoff of ridge regression with independent but non identically distributed error?

I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather ...
0
votes
0answers
22 views

R: Plot of the relationship between lambda values and coefficients in ridge regression

I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients: ...
0
votes
0answers
14 views

R: Interpreting optimal values of lambda for ridge regression the using both default and pre-defined results

I am trying to create a ridge regression model and I am currently estimating the optimal value for lambda. ...
14
votes
3answers
1k views

Why the Ridge Regression is NOT scale-invariant?

In the Element of Statistical Learning, Chapter 3, we know that the linear regression is scale-invariant since the scale matrix for coefficient will be canceled eventually, but the Ridge regression ...
1
vote
0answers
16 views

How to prove that optimal solution for ridge regression can be expressed in a following form? [closed]

We know that for ridge regression: $$\beta = (X^TX + \lambda I)^{-1}X^Ty.$$ How can I prove that the optimal solution for beta can be expressed as $$\beta = X^T*V.$$ for some V
1
vote
1answer
40 views

Ridge regression and distribution of estimate?

When OLS overfits observed data, does it give skewed distribution of estimates?
1
vote
0answers
25 views

What is the recommend function for Ridge regression [duplicate]

The following question is an answer for why lm.ridge and glmnet results are different and how to solve that. My question is ...
0
votes
0answers
8 views

What are the cv.glmnet coefficients?

Let's say one performs a ridge logistic regression with cv.glmnet with standardized variables in R. Here, the outcome of the model is given in coefficients. ...
0
votes
0answers
16 views

When using regularization wouldnt it make all parameters very small? [duplicate]

In regularization, we add square of thetas multiplied by lambda(excluding theta_0). The value of lambda is high because values of theta should be close to zero to neglect the value of its associated ...
2
votes
0answers
33 views

Effect of L2 regularization on Linear Regression

I am starting with L2 regularization on linear regression. Case without regularization: objective function $(Xw-y)^T(Xw-y)$ parameters vector $w= (X^TX)^{-1}X^Ty.$ Case with regularization: ...
0
votes
0answers
17 views

How can I get p-values from a regularized GLMM?

I have a dataset containing information about patients in a hospital, with the following variables: Status for a certain disease (binary outcome) Hundreds of continuous biomarkers A few variables for ...
0
votes
1answer
47 views

How to choose the correct dataset transformation

I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction ...
2
votes
0answers
26 views

How does ridge regression actually utilize principal components?

This is what I understand so far: Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a ...
2
votes
0answers
22 views

Does the test error for ridge regression include the regularization term or not? [closed]

When you compute the test error for ridge regression, is it typically computed with the regularization term in it?
0
votes
0answers
27 views

Cross validation for kernel regression

I've been reading about the Kernel trick, where we we can obtain a prediction by calculating: $\hat{y} = y(K(x,x)+\lambda I_n)^{−1} K(x,\hat{x})$, where $K(x,z) = (1+xz)^p$. If I want to tune the ...
2
votes
0answers
20 views

Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?

I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
0
votes
0answers
15 views

Does the number of variables change the outcome of an AUC value?

I have three different datasets, and I want to compare their predictive power of weather or not condition X will occur. In dataset one, we only have three ...
1
vote
0answers
15 views

Can regularisation reduce the accuracy in the validation test?

I am constructing a CNN neural network with TensorFlow. I have run two versions of the CNN, one of them without regularization and the other with a kernel regularizer $L^2$ in each convolutional layer....
1
vote
0answers
17 views

Weight regularisation in CNN

I am trying to understand the concept of weight regularisation in CNN. I know that in dense layer with weight $w$ it corresponds to finding: $$ \mathbf{w}^{*}=\underset{\mathbf{w}}{\arg \operatorname{...
2
votes
1answer
57 views

Intuitive understanding and practice of Group Lasso

I have been searching for a straight answer to clarify myself about the use of dummy variables in Lasso regression. I understand why we need to group them but I could not find any clear information on ...
2
votes
0answers
51 views

Why Ridge and Lasso regression is returning almost identical results to Linear Regression [closed]

I was trying to compare Ridge, Lasso and Linear Regression models to each other. I am using a subset of the Ames housing dataset. Here is a link to an already preprocessed dataset that I am using. The ...
2
votes
1answer
36 views

How is the standard error calculated for the `lambda.1se` output in the cv.glmnet function?

I understand that lambda.1se is the largest lambda that gives MSE within one standard error of the minimum MSE. But how is the standard error calculated exactly.
0
votes
0answers
17 views

Implementing L1 and L2 penalty in sequential coordinate descent least squares

I'm trying to implement L1 and L2 regularization in a fast RcppArmadillo function for non-negative least squares. The function below is adapted from the NNLM R package, receives an initial value for <...
0
votes
0answers
19 views

Why wouldn't you perform PCA before performing ridge regression on highly correlated parameters?

I'm trying to wrap my head around the L2 regularization component in ridge regression, to build a model on noisy, correlated data. I understand the $\lambda$ introduces a penalty for high bias during ...
0
votes
0answers
12 views

Standardized inputs in KRR

Say we have N observations of a function f and that one wishes to obtain an approximation of f by using Kernel Ridge Regression. I read that it was recommended to standardize the inputs. So, when one ...
1
vote
1answer
27 views

Numerical solution to the constrained ridge regression

The constrained ridge regression problem is of the form: $\arg\min_{\|\beta\|_2\le t}\|X\beta-y\|_2$. Given a matrix $X$, a vector $y$ and the constrain parameter $t$, how do you solve it numerically? ...
0
votes
1answer
40 views

Bias of Ridge Estimates in Regression

For a given ridge parameter, ridge estimates minimize the sum of squared predictions subject to an inequality constraint. Are the ridge estimates biased regardless of whether the aforementioned ...
7
votes
1answer
959 views

Why regularization parameter called as lambda in theory and alpha in python?

I was learning about regularization and came across the term called regularization parameter. I see that it is called lambda in ...
0
votes
1answer
66 views

Why feature selection using `L1` and not using `L2` norm? [duplicate]

I read a tutorial here. In which, I came across the below plots I read an explanation quoted ...
1
vote
0answers
126 views

L1 vs L2 norm - Circle and Diamond [duplicate]

I am new to ML and recently came across the L1 and L2 norm. The tutorials that I read here and here show some circle and diamond ...
5
votes
0answers
131 views

Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?

I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I ...
1
vote
1answer
28 views

In which cases should I split the data in training set and test set [closed]

I am taking a course on machine learning and in one problem I should perform a Ridge regression to fit some given data to a known model. I was wondering if, in this case, there are any advantage in ...
3
votes
1answer
45 views

On a Ridge regression-like problem

I am trying to implement a sort of Ridge regression for the following problem \begin{equation} y = a^\top X b, \end{equation} where $y\in \mathbb{R}$, $a\in\mathbb{R}^M$, $b\in\mathbb{R}^N$ and $X\in\...
0
votes
0answers
28 views

Extreme learning machine with ridge regression

I'm new to extreme learning machine (ELM) which is a single layer feedforward neural network. I'm trying to write a ridge version of the classical ELM. But this Ridge-ELM confuses me. I think the ...
1
vote
3answers
317 views

How to select the best feature set from Ridge regression?

I have applied L2 regularization on my features and have got coefficient values as below(hiding the column name due to client work: I am unsure about what all features should I choose? Should I ...
0
votes
0answers
20 views

Interpreting coefficient values in lasso or ridge regression

I am doing feature selection on a dataset using Lasso and Ridge regression. I have already standardized the Features, except the Target feature. I am new to this, so just wanted to check that after ...
2
votes
0answers
22 views

Intuition behind the degrees of freedom in ridge regression

I'm reading through the ESL book and I'm on the part of ridge regression where the effective degrees of freedom are defined $$ df(\lambda) = tr(X(X'X + \lambda I)^{-1}X') = \sum_{j=1}^p{\frac{d_j^2}{...
0
votes
1answer
21 views

Ridge Regression - Advice on Modeling Sales Data

I am looking to use ridge regression to predict end of quarter sales revenue. My features are sales pipeline and revenue booked quarter to date. As the quarter progresses sales pipeline will ...

1
2 3 4 5
14