Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
654
questions
0
votes
0answers
13 views
When to use ridge regression or lasso rather than elastic net?
If one has no ex-ante information about what the L1-ratio hyperparameter should be in the context of elastic net regularization, when should one instead use lasso or ridge? This ratio is referred to ...
2
votes
0answers
39 views
Can we estimate independent parameters when $p > n$?
I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals.
I know that when we estimate fixed effects, the number of ...
0
votes
2answers
97 views
Orthogonality of columns of the augmented design matrix for ridge regression
In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
1
vote
1answer
34 views
Ridge and Lasso in GLMs
In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others ...
0
votes
0answers
28 views
Inner Product RKHS and regression
Say you want to solve the kernel ridge regression as follows:
$\min\limits_{f\in H_K} \left[\lambda||f||^{2}_{H_K} + \sum\limits_{i=1}^{N}(y_i - f(x_i))^2\right]$, where $\lambda > 0$.
We know how ...
1
vote
1answer
59 views
Why does ridge regression only have one hyperparameter $\lambda$?
Ridge Regression objective$$\underset{\beta}{\text{min}}
\sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$
SVM primal problem:
$$\begin{align}
\max_{\mathbf{\alpha}} \quad &\min_{\...
5
votes
1answer
103 views
When there are more variables than observations do shrinkage methods (such as Ridge and Lasso) always find a solution?
Assume we have $n$ observations and $p$ explanatory variables we want to model. To apply ridge regression, we choose a constraint parameter $\lambda \geq 0$ and estimate the coefficients $\beta_i$ ...
2
votes
1answer
80 views
Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?
I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
0
votes
0answers
47 views
Why does area under curve not change from 0.5?
I have performed a ridge logistic regression with glmnet and now I look at the performance metric AUC. The script is:
...
1
vote
1answer
35 views
How to optimise penalty parameter in ridge regression using AIC
So I know for a ridge regression model, we need to find an optimal $\lambda$ value.
I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
1
vote
1answer
28 views
LassoCV vs RidgeCV in Python — why are their default number of folds different?
In https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html, it says that LassoCV defaults to 5 folds.
In https://scikit-learn.org/stable/...
1
vote
0answers
12 views
How do the training and cross validation mean squared error curves behave as a function of $\lambda$?
I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
2
votes
2answers
72 views
Bias variance tradeoff of ridge regression with independent but non identically distributed error?
I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather ...
0
votes
0answers
22 views
R: Plot of the relationship between lambda values and coefficients in ridge regression
I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients:
...
0
votes
0answers
14 views
R: Interpreting optimal values of lambda for ridge regression the using both default and pre-defined results
I am trying to create a ridge regression model and I am currently estimating the optimal value for lambda.
...
14
votes
3answers
1k views
Why the Ridge Regression is NOT scale-invariant?
In the Element of Statistical Learning, Chapter 3, we know that the linear regression is scale-invariant since the scale matrix for coefficient will be canceled eventually, but the Ridge regression ...
1
vote
0answers
16 views
How to prove that optimal solution for ridge regression can be expressed in a following form? [closed]
We know that for ridge regression:
$$\beta = (X^TX + \lambda I)^{-1}X^Ty.$$
How can I prove that the optimal solution for beta can be expressed as $$\beta = X^T*V.$$ for some V
1
vote
1answer
40 views
Ridge regression and distribution of estimate?
When OLS overfits observed data, does it give skewed distribution of estimates?
1
vote
0answers
25 views
What is the recommend function for Ridge regression [duplicate]
The following question is an answer for why lm.ridge and glmnet results are different and how to solve that. My question is ...
0
votes
0answers
8 views
What are the cv.glmnet coefficients?
Let's say one performs a ridge logistic regression with cv.glmnet with standardized variables in R. Here, the outcome of the model is given in coefficients.
...
0
votes
0answers
16 views
When using regularization wouldnt it make all parameters very small? [duplicate]
In regularization, we add square of thetas multiplied by lambda(excluding theta_0). The value of lambda is high because values of theta should be close to zero to neglect the value of its associated ...
2
votes
0answers
33 views
Effect of L2 regularization on Linear Regression
I am starting with L2 regularization on linear regression.
Case without regularization:
objective function $(Xw-y)^T(Xw-y)$
parameters vector $w= (X^TX)^{-1}X^Ty.$
Case with regularization:
...
0
votes
0answers
17 views
How can I get p-values from a regularized GLMM?
I have a dataset containing information about patients in a hospital, with the following variables:
Status for a certain disease (binary outcome)
Hundreds of continuous biomarkers
A few variables for ...
0
votes
1answer
47 views
How to choose the correct dataset transformation
I'm doing a project using the California Housing Price dataset from Kaggle. The objectetive of the project is to implement from scratch the Ridge Regression algorithm, apply it the to the prediction ...
2
votes
0answers
26 views
How does ridge regression actually utilize principal components?
This is what I understand so far:
Let $\mathbf X$ be the centered $n \times p$ predictor matrix and
consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a ...
2
votes
0answers
22 views
Does the test error for ridge regression include the regularization term or not? [closed]
When you compute the test error for ridge regression, is it typically computed with the regularization term in it?
0
votes
0answers
27 views
Cross validation for kernel regression
I've been reading about the Kernel trick, where we we can obtain a prediction by calculating: $\hat{y} = y(K(x,x)+\lambda I_n)^{−1} K(x,\hat{x})$, where $K(x,z) = (1+xz)^p$. If I want to tune the ...
2
votes
0answers
20 views
Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?
I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
0
votes
0answers
15 views
Does the number of variables change the outcome of an AUC value?
I have three different datasets, and I want to compare their predictive power of weather or not condition X will occur.
In dataset one, we only have three ...
1
vote
0answers
15 views
Can regularisation reduce the accuracy in the validation test?
I am constructing a CNN neural network with TensorFlow. I have run two versions of the CNN, one of them without regularization and the other with a kernel regularizer $L^2$ in each convolutional layer....
1
vote
0answers
17 views
Weight regularisation in CNN
I am trying to understand the concept of weight regularisation in CNN. I know that in dense layer with weight $w$ it corresponds to finding:
$$
\mathbf{w}^{*}=\underset{\mathbf{w}}{\arg \operatorname{...
2
votes
1answer
57 views
Intuitive understanding and practice of Group Lasso
I have been searching for a straight answer to clarify myself about the use of dummy variables in Lasso regression. I understand why we need to group them but I could not find any clear information on ...
2
votes
0answers
51 views
Why Ridge and Lasso regression is returning almost identical results to Linear Regression [closed]
I was trying to compare Ridge, Lasso and Linear Regression models to each other. I am using a subset of the Ames housing dataset. Here is a link to an already preprocessed dataset that I am using. The ...
2
votes
1answer
36 views
How is the standard error calculated for the `lambda.1se` output in the cv.glmnet function?
I understand that lambda.1se is the largest lambda that gives MSE within one standard error of the minimum MSE. But how is the standard error calculated exactly.
0
votes
0answers
17 views
Implementing L1 and L2 penalty in sequential coordinate descent least squares
I'm trying to implement L1 and L2 regularization in a fast RcppArmadillo function for non-negative least squares.
The function below is adapted from the NNLM R package, receives an initial value for <...
0
votes
0answers
19 views
Why wouldn't you perform PCA before performing ridge regression on highly correlated parameters?
I'm trying to wrap my head around the L2 regularization component in ridge regression, to build a model on noisy, correlated data.
I understand the $\lambda$ introduces a penalty for high bias during ...
0
votes
0answers
12 views
Standardized inputs in KRR
Say we have N observations of a function f and that one wishes to obtain an approximation of f by using Kernel Ridge Regression. I read that it was recommended to standardize the inputs. So, when one ...
1
vote
1answer
27 views
Numerical solution to the constrained ridge regression
The constrained ridge regression problem is of the form:
$\arg\min_{\|\beta\|_2\le t}\|X\beta-y\|_2$. Given a matrix $X$, a vector $y$ and the constrain parameter $t$, how do you solve it numerically?
...
0
votes
1answer
40 views
Bias of Ridge Estimates in Regression
For a given ridge parameter, ridge estimates minimize the sum of squared predictions subject to an inequality constraint.
Are the ridge estimates biased regardless of whether the aforementioned ...
7
votes
1answer
959 views
Why regularization parameter called as lambda in theory and alpha in python?
I was learning about regularization and came across the term called regularization parameter.
I see that it is called lambda in ...
0
votes
1answer
66 views
Why feature selection using `L1` and not using `L2` norm? [duplicate]
I read a tutorial here. In which, I came across the below plots
I read an explanation quoted ...
1
vote
0answers
126 views
L1 vs L2 norm - Circle and Diamond [duplicate]
I am new to ML and recently came across the L1 and L2 norm.
The tutorials that I read here and here show some circle and diamond ...
5
votes
0answers
131 views
Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?
I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I ...
1
vote
1answer
28 views
In which cases should I split the data in training set and test set [closed]
I am taking a course on machine learning and in one problem I should perform a Ridge regression to fit some given data to a known model. I was wondering if, in this case, there are any advantage in ...
3
votes
1answer
45 views
On a Ridge regression-like problem
I am trying to implement a sort of Ridge regression for the following problem
\begin{equation}
y = a^\top X b,
\end{equation}
where $y\in \mathbb{R}$, $a\in\mathbb{R}^M$, $b\in\mathbb{R}^N$ and $X\in\...
0
votes
0answers
28 views
Extreme learning machine with ridge regression
I'm new to extreme learning machine (ELM) which is a single layer feedforward neural network. I'm trying to write a ridge version of the classical ELM. But this Ridge-ELM confuses me. I think the ...
1
vote
3answers
317 views
How to select the best feature set from Ridge regression?
I have applied L2 regularization on my features and have got coefficient values as below(hiding the column name due to client work:
I am unsure about what all features should I choose? Should I ...
0
votes
0answers
20 views
Interpreting coefficient values in lasso or ridge regression
I am doing feature selection on a dataset using Lasso and Ridge regression. I have already standardized the Features, except the Target feature. I am new to this, so just wanted to check that after ...
2
votes
0answers
22 views
Intuition behind the degrees of freedom in ridge regression
I'm reading through the ESL book and I'm on the part of ridge regression where the effective degrees of freedom are defined
$$
df(\lambda) = tr(X(X'X + \lambda I)^{-1}X') = \sum_{j=1}^p{\frac{d_j^2}{...
0
votes
1answer
21 views
Ridge Regression - Advice on Modeling Sales Data
I am looking to use ridge regression to predict end of quarter sales revenue. My features are sales pipeline and revenue booked quarter to date. As the quarter progresses sales pipeline will ...
