Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
0
votes
0answers
39 views

Can we exclude non-significant variables (p>0.05) in ridge regression?

As far as I know, variables with p>0.05 are non-significant for the regression model. I found similar questions here about p-value like: What is the meaning of p values and t values in statistical ...
0
votes
0answers
66 views

Feature Selection for Ridge Regression

There is a closed-form equation for computing the optimal weight matrix $\mathbf{W}_{\text{optimal}}$ according to the feature matrix $\mathbf{\Phi}$ and the target matrix $\mathbf{Z}$ for ridge ...
2
votes
0answers
89 views

When to use LSTM vs Lasso/Ridge Regression vs ARIMA?

I have a set of N time series and want to make predictions about the future values of these N elementary time signals. From a first rough analysis, I can say that at a given moment in time, the N ...
3
votes
2answers
149 views

glmnet for binary outcomes: Why is “%Dev” inversely correlated with lambda?

I am new to glmnet but would like to apply it to a dataset with binary outcomes. Can you please clarify a few questions for me? Below are the codes and data setup <...
1
vote
0answers
30 views

Should Kernel Ridge Regression with linear kernel yield same results as Ridge regression?

I'm comparing the performance of different regressors from scikit-learn for fitting some data. I would have expected that Ridge regression and Kernel Ridge regression both yield the same model/...
0
votes
0answers
36 views

What is the impact of the weight decay on self normalizing neural network with selu activations?

So there is a regularization technique called weight decay performing thikonov regularization (or in statistics community ridge regression). There is also a (lets say new) approach for neural ...
1
vote
1answer
51 views

How should I consider the signs of the beta weights in a composite?

I have some biomarkers ($X_1, \ldots, X_5$) and I want to model an outcome ($Y$) using these biomarkers. The biomarkers are correlated. So I decided to use a ridge regression to stabilize the ...
7
votes
2answers
235 views

Is there a canonical example of when ridge outperforms lasso?

Can someone please give me an example of when ridge would out perform lasso? Won't lasso do better in most circumstances? If a regressor has a large coefficient, that means the regressor is a good ...
1
vote
0answers
30 views

Effect of log transformation or standardization of a regressor in the filtering step

We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
0
votes
1answer
43 views

What is the difference between “variables of interest” and “variables from which lasso selects” in Lasso?

I'm getting started with regularization models and I notice that Lasso requires three inputs, a dependent variable and then two sets of what I assume as independent variables, one which are of ...
2
votes
0answers
48 views

Getting started with regularization (Lasso)

I've got a small data set of 55 observations with a binary outcome variable of which only 11 are 1's and the rest are 0's. I was wondering if Lasso was a useful tool to predict my outcome here and if ...
3
votes
0answers
161 views

How to consider interaction terms in the ridge / lasso / elastic net?

I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
1
vote
1answer
69 views

Comparison of regression models in terms of the importance of variables

I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values ​​obtained are not on ...
16
votes
4answers
2k views

In regression, why not use regularization by default?

I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
2
votes
0answers
21 views

What are the available method that can alleviate the overfitting problem in traditional OLS problem, but still can get a linear fitting?

Recently, I have read the paper https://static1.squarespace.com/static/56def54a45bf21f27e160072/t/5a0d0673419202ef1b2259f2/1510803060244/The_Sampling_Error_in_Estimates_of_Mean-...
6
votes
1answer
538 views

AIC and its degrees of freedom for linear regression models

I have a dataset $S$ with $D$ features and three fitted linear regression models: Model1. Ridge regression that is fitted on all $D$ features from $S$. Model2. Ridge regression that is fitted on some $...
1
vote
1answer
98 views

False positive/negative rate in ridge and lasso regressions

I have a confusion matrix of true and estimated $\boldsymbol{\beta}$ vectors of lasso and ridge models from a replicate of a simulation study, say. The following tables illustrate the scenario. $$\...
4
votes
0answers
203 views

Results of cv.glmnet in R versus RidgeCV in scikit-learn

I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R). First a few things to note: ...
0
votes
0answers
13 views

Getting different values for MSE using anova(lm(y~.)) and mean(residuals(fit)^2)

Using this dataset of gas mileage for different cars I've been asked to run a ridge regression using $\frac{p*\sigma^2}{\beta'\beta}$ as the k-value. I've been told $\sigma^2 = MSE$ $p =$ ...
1
vote
1answer
49 views

Robust regression with M-estimators

I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator: Is it possible/common to combine these with regularization terms, such ...
0
votes
0answers
63 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
1
vote
1answer
176 views

Poor performance on Regularized models

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've ...
1
vote
0answers
72 views

Efficiency of ridge regression in under determined systems

Imagine an underdetermined linear system, composed of N (continuous) labels and N samples, each have P features (with N < P): $$\hat{\textbf{Y}}_{N \times 1} = \textbf{X}_{N \times P} \textbf{W}_{...
2
votes
1answer
151 views

Why cant Ridge Regression benift from negative lamda? [duplicate]

in Rigid regression, we generally set a positive Lambda for regularization to get a less Residual. Why cant we have a negative Lambda in a regularization if we can benefit from it?
1
vote
1answer
116 views

Statistical library for orthogonal distance regression with a ridge penalty?

There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
0
votes
1answer
97 views

l2 lambdas in Keras.regularizers [closed]

Is the value supplied to the shrinkage regularizers (l1 and l2) in Keras the inverse of the lambda coefficient? e.g. ...
0
votes
1answer
145 views

Derivation of ridge regression for multi-value-target vectors

At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$. However, now I need to ...
3
votes
1answer
61 views

Why glmnet 's $\lambda$ value is so small? Does it strictly implement the loss function under the hood?

I am running a glmnet fit with 1200000 samples. According to the glmnet doc, $\lambda$ value is the coefficient controlling how much the regularization term contributes to the total loss function. ...
2
votes
2answers
139 views

Simplifying the Matrix Form of the Solution to Ridge Regression

I'm trying to understand how to obtain the solution to an objective function by solving for the parameter vector $\theta$ in ridge regression. I found an example here from Naomi which takes an example ...
1
vote
1answer
72 views

High odds ratio for composite score created by ridge regression

This question is a follow up to one of my previous questions asked on this site. The goal was to create a composite score for biomarkers related to a binary outcome and then use that in a regression ...
2
votes
0answers
308 views

Shapley value vs ridge regression

My goal is to get the feature importance for multiple regression. I have a data set with some multicollinearity. I found two methods to solve this problem. The first one is the Shapley value. ...
5
votes
1answer
415 views

L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....
1
vote
2answers
2k views

Implementing ridge regression in python

I was trying to implement ridge regression in python. I implemented the following code: ...
1
vote
0answers
7 views

Does the initial distribution of data have any affect on which regularization parameter can work well?

In scenarios when we want to know why performance of a predicting linear regression model when using L1 regularization has outperformed with the case that we have used L2 regularization, I wonder ...
0
votes
0answers
257 views

Ridge regression is similar to Linear regression [duplicate]

I can not see any difference between Ridge Regression and Linear Regression MY understanding, The point of ridge Regression is based on the training data we find the best line that fits training ...
1
vote
1answer
96 views

Understanding concepts of regularization

I am trying to understand regularization in machine learning. But, I do not understand some fundamental concepts in this topic, could you please explain? A model that has high variance, captures ...
0
votes
1answer
2k views

R Ridge Regression: Choosing best lambda

I am doing ridge regression with Mass package and stuck with the problem trying to find the best lambda. I know that it should look somehow like thi ...
6
votes
3answers
207 views

Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
2
votes
0answers
29 views

Variance of $\hat{\beta}$ in Ridge Regression

If you are using ridge regression, what happens to the variance of your parameter estimates relative to regular regression? My intuition is telling me that it would decrease because you are doing a ...
4
votes
2answers
100 views

variance of the square of the bias on linear regression

Basic setting let the linear model be: $$ \mathbf{y}=\mathbf{X\beta}+\epsilon $$ where $\epsilon \sim N(0,\sigma^2\mathbf{I}_n)$ $n$ is the number of samples $p$ is the number of attributes. $\...
0
votes
3answers
228 views

Should one drop independent variables if they don't have linear relationship with the response variable?

I am building a linear regression model using Ridge regression. Some of the independent variables don't have linear relationships with the dependent variable. I've tried to do data transformations on ...
0
votes
0answers
61 views

Elastic net regression with uneven penalties for predictors

For a regression model where you are certain that y that depends on some predictors but are agnostic about whether some other predictors should enter, how should you incorporate this prior information?...
0
votes
0answers
85 views

Should we penalize dummy variables? [duplicate]

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
0
votes
1answer
566 views

Why is Lasso and Ridge not giving better results than OLS?

I am trying to find an example in which Lasso and Ridge regression are doing better than simple OLS. I am trying to run the Boston example that appears in the MASS library in R. The dependent ...
0
votes
0answers
56 views

Ridge vs. Lasso vs. Elastic Net [duplicate]

I have a theoretical question. I was reading about ridge regression, lasso and the elastic net, and is very impressed. One thing is not quite clear to me. I would like to know when should I use each ...
1
vote
1answer
32 views

In what situations would it be appropriate to use Ridge Regression vs Multiple Regression and vice versa?

My understanding is that regularization will generally help prediction tasks. What about for situations where we want to conduct a study to understand the effect of a specific predictor on the ...
16
votes
2answers
536 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
4
votes
1answer
193 views

Rademacher Bound, An Alternative to Cross Validation for Ridge?

Below is a theorem from the book "Foundations of Machine Learning". It specifies the generalization bounds for Kernel Ridge Regression by making use of the Rademacher Complexity on linear models. $R(...
4
votes
1answer
166 views

Alternatives to Pre-Scaling Predictors in Lasso/Ridge Regression?

In lasso/ridge regression it's often recommended to scale predictors $X$ before estimation so that the coefficient estimates $\hat{\beta}$ will be invariant to the scale of predictors $X$. Q: Is ...
6
votes
2answers
327 views

One-to-one correspondence between penalty parameters of equivalent formulations of penalised regression methods

Ridge, LASSO and Elastic Net are three very popular methods of penalised regressions. All of these have more than one formulations. For example, two formulations for Ridge are: minimise $\lVert Y - X ...

1 2
3
4 5
13