Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
522
questions
1
vote
0answers
8 views
Should Kernel Ridge Regression with linear kernel yield same results as Ridge regression?
I'm comparing the performance of different regressors from scikit-learn for fitting some data.
I would have expected that Ridge regression and Kernel Ridge regression both yield the same model/...
0
votes
0answers
12 views
What is the impact of the weight decay on self normalizing neural network with selu activations?
So there is a regularization technique called weight decay performing thikonov regularization (or in statistics community ridge regression).
There is also a (lets say new) approach for neural ...
0
votes
0answers
12 views
How should I consider the signs of the beta weights in a composite?
I had covariates (say, $X_1$) and some biomarkers ($X_2, \ldots, X_5$) and I wanted to model an outcome ($Y$) using these biomarkers. The biomarkers are correlated. So I decided to use a ridge ...
5
votes
2answers
148 views
Is there a canonical example of when ridge outperforms lasso?
Can someone please give me an example of when ridge would out perform lasso?
Won't lasso do better in most circumstances? If a regressor has a large coefficient, that means the regressor is a good ...
0
votes
0answers
15 views
Effect of log transformation or standardization of a regressor in the filtering step
We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
0
votes
1answer
24 views
What is the difference between “variables of interest” and “variables from which lasso selects” in Lasso?
I'm getting started with regularization models and I notice that Lasso requires three inputs, a dependent variable and then two sets of what I assume as independent variables, one which are of ...
2
votes
0answers
36 views
Getting started with regularization (Lasso)
I've got a small data set of 55 observations with a binary outcome variable of which only 11 are 1's and the rest are 0's.
I was wondering if Lasso was a useful tool to predict my outcome here and if ...
3
votes
0answers
20 views
How to consider interaction terms in the ridge / lasso / elastic net?
I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
1
vote
1answer
23 views
Comparison of regression models in terms of the importance of variables
I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values obtained are not on ...
15
votes
4answers
1k views
In regression, why not use regularization by default?
I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
2
votes
0answers
18 views
What are the available method that can alleviate the overfitting problem in traditional OLS problem, but still can get a linear fitting?
Recently, I have read the paper https://static1.squarespace.com/static/56def54a45bf21f27e160072/t/5a0d0673419202ef1b2259f2/1510803060244/The_Sampling_Error_in_Estimates_of_Mean-...
4
votes
1answer
63 views
AIC and its degrees of freedom for linear regression models
I have a dataset $S$ with $D$ features and three fitted linear regression models:
Model1. Ridge regression that is fitted on all $D$ features from $S$.
Model2. Ridge regression that is fitted on some $...
1
vote
1answer
32 views
False positive/negative rate in ridge and lasso regressions
I have a confusion matrix of true and estimated $\boldsymbol{\beta}$ vectors of lasso and ridge models from a replicate of a simulation study, say. The following tables illustrate the scenario.
$$\...
4
votes
0answers
42 views
Results of cv.glmnet in R versus RidgeCV in scikit-learn
I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R).
First a few things to note:
...
0
votes
0answers
12 views
Getting different values for MSE using anova(lm(y~.)) and mean(residuals(fit)^2)
Using this dataset of gas mileage for different cars I've been asked to run a ridge regression using $\frac{p*\sigma^2}{\beta'\beta}$ as the k-value.
I've been told
$\sigma^2 = MSE$
$p =$ ...
1
vote
1answer
30 views
Robust regression with M-estimators
I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator:
Is it possible/common to combine these with regularization terms, such ...
0
votes
0answers
25 views
Ridge Regression worse results with more feature. Does it make sense?
PREMISE
I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data.
I took care ...
1
vote
1answer
46 views
Poor performance on Regularized models
I'm trying to build a simple model to predict the price of a cab ride, using features
such as hour, source, destination, car model, distance, and
weather features such as pressure and humidity.
I've ...
1
vote
0answers
38 views
Efficiency of ridge regression in under determined systems
Imagine an underdetermined linear system, composed of N (continuous) labels and N samples, each have P features (with N < P):
$$\hat{\textbf{Y}}_{N \times 1} = \textbf{X}_{N \times P} \textbf{W}_{...
2
votes
1answer
106 views
Why cant Ridge Regression benift from negative lamda? [duplicate]
in Rigid regression, we generally set a positive Lambda for regularization to get a less Residual. Why cant we have a negative Lambda in a regularization if we can benefit from it?
0
votes
0answers
17 views
Estimating coefficients of a linear model with collinear dependent variables that have errors with known variances
I want to estimate the coefficients $\beta$ of the linear model
$Y=\beta X$
from observations of $(Y_i,X_i), i=1\ldots n$, where $X$ is multidimensional.
Two problems:
All variables have been ...
1
vote
1answer
60 views
Statistical library for orthogonal distance regression with a ridge penalty?
There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
0
votes
0answers
10 views
Variance of ridge regression estimator under perturbed data
In section 1.3 Ridge Regression as Perturbation of the notes the author comes up with the following ridge estimator
$$
\widehat{\boldsymbol{\beta}}=\left[(\mathbf{X}+\mathbf{W})^{T}(\mathbf{X}+\mathbf{...
0
votes
0answers
13 views
Regression training and testing error
Let’s say we fit a linear regression. What does the correlation between its training error and testing error say about the model, its performance or the data? What does a very low or very high ...
0
votes
1answer
30 views
l2 lambdas in Keras.regularizers [closed]
Is the value supplied to the shrinkage regularizers (l1 and l2) in Keras the inverse of the lambda coefficient?
e.g. ...
0
votes
0answers
12 views
How to do cross validation in ODE models with more predicted than measured time courses?
I have an ODE model of biochemical reactions with 37 state variables and 88 strictly positive parameters. Unfortunately, I can only expect to get time course measurements of about 10 state variables (...
0
votes
1answer
62 views
Derivation of ridge regression for multi-value-target vectors
At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$.
However, now I need to ...
0
votes
0answers
8 views
How To Feed Un-Scaled Data Into a Model That Was Trained on Scaled Data
I have a data frame that contains time series data. I split the dataframe into test and train. I want to prevent leakage so I split the data frame before doing any scaling.
On the train data set, I ...
3
votes
1answer
44 views
Why glmnet 's $\lambda$ value is so small? Does it strictly implement the loss function under the hood?
I am running a glmnet fit with 1200000 samples.
According to the glmnet doc, $\lambda$ value is the coefficient controlling how much the regularization term contributes to the total loss function.
...
0
votes
0answers
23 views
Estimate variance of parameters in non-linear ridge regression
I am basically estimating the shape of an unknown function f(x) from multi-dimensional chemical reaction data by estimating the most likely function values $f$ on a grid $x$ with kernel regression. To ...
0
votes
0answers
43 views
Solving ridge regression for p >> n case using dual algorithm with or without nonnegativity constraints
I was reading the paper
"Efficient Regularized Regression with
L0 Penalty for Variable Selection and Network Construction"
in which iterated ridge regression is used to solve L0 penalized regression ...
1
vote
1answer
33 views
High odds ratio for composite score created by ridge regression
This question is a follow up to one of my previous questions asked on this site. The goal was to create a composite score for biomarkers related to a binary outcome and then use that in a regression ...
0
votes
0answers
43 views
Comparing Ridge and Lasso Regression [duplicate]
I was thinking about main differences between ridge and lasso introducing a $\ell^2$ and $\ell^1$ penalty term respectively. The main thing is that with ridge I will keep all my features in the end ...
2
votes
0answers
88 views
Shapley value vs ridge regression
My goal is to get the feature importance for multiple regression. I have a data set with some multicollinearity. I found two methods to solve this problem.
The first one is the Shapley value. ...
2
votes
1answer
194 views
L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)
I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve.
I have successfully run both glmnet() and cv....
0
votes
0answers
74 views
Mathematical proof of how L1 and L2 regularization work [duplicate]
How do you mathematically prove that L1 regularization makes weights sparse but L2 regularization does not?
1
vote
2answers
417 views
Implementing ridge regression in python
I was trying to implement ridge regression in python. I implemented the following code:
...
0
votes
0answers
10 views
What is the meaning of assuming a special prior on regularization method
I have heard/read that L1 regularization assumes Laplacian prior, however L2 regularization assumes Gaussian prior.
But what exactly "assume" mean here? How does it work? How do each of these ...
1
vote
0answers
6 views
Does the initial distribution of data have any affect on which regularization parameter can work well?
In scenarios when we want to know why performance of a predicting linear regression model when using L1 regularization has outperformed with the case that we have used L2 regularization, I wonder ...
0
votes
0answers
48 views
Ridge regression is similar to Linear regression [duplicate]
I can not see any difference between
Ridge Regression and Linear Regression
MY understanding, The point of ridge Regression is based on the training data we find the best line that fits training ...
0
votes
1answer
56 views
Understanding concepts of regularization
I am trying to understand regularization in machine learning. But, I do not understand some fundamental concepts in this topic, could you please explain?
A model that has high variance, captures ...
0
votes
1answer
681 views
R Ridge Regression: Choosing best lambda
I am doing ridge regression with Mass package and stuck with the problem trying to find the best lambda.
I know that it should look somehow like thi
...
0
votes
0answers
17 views
Intuition for nonmonotonicity of coefficient paths in ridge regression
Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
2
votes
0answers
23 views
Variance of $\hat{\beta}$ in Ridge Regression
If you are using ridge regression, what happens to the variance of your parameter estimates relative to regular regression?
My intuition is telling me that it would decrease because you are doing a ...
0
votes
0answers
23 views
Equivalence of two optimization problems [duplicate]
Ridge regression problem:
$\sum_i^n(y_i-\beta_0-\beta_{1i}-\beta_{2i})^2 \longrightarrow min_{\beta}$
$s.t. \sum_i^p\beta_i^2 \leq c$
$\sum_i^n(y_i-\beta_0-\beta_{1i}-\beta_{2i})^2 + \lambda(\sum_i^...
4
votes
2answers
66 views
variance of the square of the bias on linear regression
Basic setting
let the linear model be:
$$
\mathbf{y}=\mathbf{X\beta}+\epsilon
$$
where $\epsilon \sim N(0,\sigma^2\mathbf{I}_n)$
$n$ is the number of samples
$p$ is the number of attributes.
$\...
0
votes
3answers
65 views
Should one drop independent variables if they don't have linear relationship with the response variable?
I am building a linear regression model using Ridge regression. Some of the independent variables don't have linear relationships with the dependent variable. I've tried to do data transformations on ...
0
votes
0answers
32 views
Elastic net regression with uneven penalties for predictors
For a regression model where you are certain that y that depends on some predictors but are agnostic about whether some other predictors should enter, how should you incorporate this prior information?...
0
votes
0answers
32 views
Data normalization in ridge regression when there is no intercept
I would like to have a linear model without an intercept and also without the target being centered. How should my data then be normalized when using ridge regression?
If I standardized the variables ...
0
votes
0answers
63 views
Should we penalize dummy variables? [duplicate]
Using glmnet we run the following regression
cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE)
where $y$ is the response variable and $x$ is the input matrix....
