Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
4
votes
1answer
405 views
+50

L1 vs L2 stability?

See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/ The instability property of the method of least absolute deviations means that, for ...
2
votes
2answers
43 views

Simplifying the Matrix Form of the Solution to Ridge Regression

I'm trying to understand how to obtain the solution to an objective function by solving for the parameter vector $\theta$ in ridge regression. I found an example here from Naomi which takes an example ...
0
votes
0answers
12 views

Why would one want to choose lambda.1se for ridge regression in glmnet?

In R, choosing lambda.1se over lambda.min to get a more parsimonious model is common. This post (and this) also indicated that ...
1
vote
1answer
31 views

Least Square vs Shrinkage approach of fitting models

What is the difference between the Least Square and Shrinkage approach of fitting models in the context of model selection? In https://www.youtube.com/watch?v=QlyROnAjnEk the author at [0:28] instance ...
5
votes
3answers
168 views

Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
1
vote
1answer
47 views

Howe to perform ridge regression only on a subset of the variables

I am trying to code some algorithm that performs ridge-regression with penalty parameter $\lambda$ on all features except for a specific subset. Let $\mathbf{X}$ be the $n \times p$ matrix for $n$ ...
1
vote
1answer
41 views

Difference between L1 and L2 Regularization (in Lasso and Ridge Regression)

I got a more theoretical question here: I have made some research about the L2 (Ridge) and L1 (Lasso) regularizations. I know the formula, and understand the aim of those two different procedures. The ...
0
votes
0answers
22 views

Regression with pytorch with inferior results to ridge regression

I am trying to use a simple 3 layer neural net to predict a scaler output given an input of dimension 430. For my network, I use 2 layers of dimensions 600 and 80 and I use leakyReLU non-linearities. ...
2
votes
1answer
52 views

Why is linear regression not a stable algorithm?

In the paper Stability and Generalization the author defines the stability of a learning algorithm, which intuitively means that changing one sample in the sample set does not affect the outcome much. ...
0
votes
0answers
7 views

Matlab - Financial Modeling, Linear Regression with Prior

Am trying to implement this equation from the book Doing Data Science Straight Talk from the frontline, In chapter 6, page 161, equation below: From what i can tell it is pretty much an enchanced ...
0
votes
0answers
57 views

Feature Selection for Ridge Regression

There is a closed-form equation for computing the optimal weight matrix $\mathbf{W}_{\text{optimal}}$ according to the feature matrix $\mathbf{\Phi}$ and the target matrix $\mathbf{Z}$ for ridge ...
0
votes
0answers
16 views

Can we exclude non-significant variables (p>0.05) in ridge regression?

As far as I know, variables with p>0.05 are non-significant for the regression model. I found similar questions here about p-value like: What is the meaning of p values and t values in statistical ...
1
vote
0answers
33 views

When to use LSTM vs Lasso/Ridge Regression vs ARIMA?

I have a set of N time series and want to make predictions about the future values of these N elementary time signals. From a first rough analysis, I can say that at a given moment in time, the N ...
8
votes
4answers
4k views

Prove that the variance of the ridge regression estimator is less than the variance of the OLS estimator

Consider the following linear model under classical Gauss-Markov assumtions: $$Y = X\beta + e$$ where $\mathbb{E}X'e = 0$ Consider the following estimator $$\tilde\beta = \left(\sum_{i=1}^{N}x_ix_i' +...
3
votes
2answers
59 views

glmnet for binary outcomes: Why is “%Dev” inversely correlated with lambda?

I am new to glmnet but would like to apply it to a dataset with binary outcomes. Can you please clarify a few questions for me? Below are the codes and data setup <...
1
vote
0answers
14 views

Should Kernel Ridge Regression with linear kernel yield same results as Ridge regression?

I'm comparing the performance of different regressors from scikit-learn for fitting some data. I would have expected that Ridge regression and Kernel Ridge regression both yield the same model/...
0
votes
0answers
15 views

What is the impact of the weight decay on self normalizing neural network with selu activations?

So there is a regularization technique called weight decay performing thikonov regularization (or in statistics community ridge regression). There is also a (lets say new) approach for neural ...
0
votes
0answers
12 views

How should I consider the signs of the beta weights in a composite?

I had covariates (say, $X_1$) and some biomarkers ($X_2, \ldots, X_5$) and I wanted to model an outcome ($Y$) using these biomarkers. The biomarkers are correlated. So I decided to use a ridge ...
7
votes
2answers
199 views

Is there a canonical example of when ridge outperforms lasso?

Can someone please give me an example of when ridge would out perform lasso? Won't lasso do better in most circumstances? If a regressor has a large coefficient, that means the regressor is a good ...
15
votes
4answers
1k views

In regression, why not use regularization by default?

I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
0
votes
0answers
16 views

Effect of log transformation or standardization of a regressor in the filtering step

We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
0
votes
1answer
27 views

What is the difference between “variables of interest” and “variables from which lasso selects” in Lasso?

I'm getting started with regularization models and I notice that Lasso requires three inputs, a dependent variable and then two sets of what I assume as independent variables, one which are of ...
1
vote
1answer
228 views

Is there any two-stage procedure for elastic net as LASSO?

I read this post Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables? . It says the LASSO shrinkage causes the estimates of the non-zero coefficients to be biased ...
1
vote
1answer
28 views

Comparison of regression models in terms of the importance of variables

I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values ​​obtained are not on ...
2
votes
0answers
39 views

Getting started with regularization (Lasso)

I've got a small data set of 55 observations with a binary outcome variable of which only 11 are 1's and the rest are 0's. I was wondering if Lasso was a useful tool to predict my outcome here and if ...
3
votes
0answers
26 views

How to consider interaction terms in the ridge / lasso / elastic net?

I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
2
votes
1answer
502 views

R: What does train() do when it calculates ridge regression?

I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression. I will write up the two methods and then pose my question Initialize with the ...
2
votes
0answers
18 views

What are the available method that can alleviate the overfitting problem in traditional OLS problem, but still can get a linear fitting?

Recently, I have read the paper https://static1.squarespace.com/static/56def54a45bf21f27e160072/t/5a0d0673419202ef1b2259f2/1510803060244/The_Sampling_Error_in_Estimates_of_Mean-...
4
votes
1answer
98 views

AIC and its degrees of freedom for linear regression models

I have a dataset $S$ with $D$ features and three fitted linear regression models: Model1. Ridge regression that is fitted on all $D$ features from $S$. Model2. Ridge regression that is fitted on some $...
1
vote
1answer
35 views

False positive/negative rate in ridge and lasso regressions

I have a confusion matrix of true and estimated $\boldsymbol{\beta}$ vectors of lasso and ridge models from a replicate of a simulation study, say. The following tables illustrate the scenario. $$\...
4
votes
0answers
51 views

Results of cv.glmnet in R versus RidgeCV in scikit-learn

I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R). First a few things to note: ...
0
votes
0answers
12 views

Getting different values for MSE using anova(lm(y~.)) and mean(residuals(fit)^2)

Using this dataset of gas mileage for different cars I've been asked to run a ridge regression using $\frac{p*\sigma^2}{\beta'\beta}$ as the k-value. I've been told $\sigma^2 = MSE$ $p =$ ...
2
votes
0answers
849 views

How to decide whether to use Ridge Regression/LASSO/Elastic Net or Random Forest for Feature Selection?

My understanding is rudimentary and high level but it seems like Ridge Regression/LASSO/Elastic Net would be better when the data is linear and Random Forest is better when the data is nonlinear? Also ...
1
vote
1answer
31 views

Robust regression with M-estimators

I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator: Is it possible/common to combine these with regularization terms, such ...
3
votes
1answer
129 views

Is group lasso equivalent to ridge regression when there is 1 group

On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso). However in group lasso we have norm ...
0
votes
0answers
26 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
14
votes
2answers
6k views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
1
vote
1answer
53 views

Poor performance on Regularized models

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've ...
1
vote
0answers
40 views

Efficiency of ridge regression in under determined systems

Imagine an underdetermined linear system, composed of N (continuous) labels and N samples, each have P features (with N < P): $$\hat{\textbf{Y}}_{N \times 1} = \textbf{X}_{N \times P} \textbf{W}_{...
0
votes
1answer
70 views

Derivation of ridge regression for multi-value-target vectors

At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$. However, now I need to ...
1
vote
1answer
67 views

Statistical library for orthogonal distance regression with a ridge penalty?

There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
2
votes
1answer
109 views

Why cant Ridge Regression benift from negative lamda? [duplicate]

in Rigid regression, we generally set a positive Lambda for regularization to get a less Residual. Why cant we have a negative Lambda in a regularization if we can benefit from it?
0
votes
0answers
17 views

Estimating coefficients of a linear model with collinear dependent variables that have errors with known variances

I want to estimate the coefficients $\beta$ of the linear model $Y=\beta X$ from observations of $(Y_i,X_i), i=1\ldots n$, where $X$ is multidimensional. Two problems: All variables have been ...
2
votes
1answer
218 views

L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....
0
votes
0answers
11 views

Variance of ridge regression estimator under perturbed data

In section 1.3 Ridge Regression as Perturbation of the notes the author comes up with the following ridge estimator $$ \widehat{\boldsymbol{\beta}}=\left[(\mathbf{X}+\mathbf{W})^{T}(\mathbf{X}+\mathbf{...
5
votes
4answers
994 views

Rationale behind shrinking regression coefficients in Ridge or LASSO regression

I understand that with Ridge or Lasso regression we are trying to shrink regression coefficients, and we specify the amount of shrinking we need by varying alpha. But I cannot understand the intuition ...
21
votes
2answers
983 views

The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
4
votes
1answer
239 views

Ridge regression: penalizing weights corresponding to larger-scale features

In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function. On page 5, in the little section called "...
1
vote
1answer
193 views

glmnet package: “mgaussian” vs “gaussian” for $\alpha = 0$

In multiresponse Gaussian family the objective function when $\alpha = 0$: \begin{align} \frac{1}{2n}||Y-XB||_F^2 + \frac{\lambda}{2}||B||_F^2. \end{align} This can also mathematically solved as \...
0
votes
0answers
14 views

Regression training and testing error

Let’s say we fit a linear regression. What does the correlation between its training error and testing error say about the model, its performance or the data? What does a very low or very high ...

1 2 3 4 5 11