Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
530
questions
1
vote
1answer
44 views
Howe to perform ridge regression only on a subset of the variables
I am trying to code some algorithm that performs ridge-regression with penalty parameter $\lambda$ on all features except for a specific subset.
Let $\mathbf{X}$ be the $n \times p$ matrix for $n$ ...
1
vote
1answer
34 views
Difference between L1 and L2 Regularization (in Lasso and Ridge Regression)
I got a more theoretical question here: I have made some research about the L2 (Ridge) and L1 (Lasso) regularizations. I know the formula, and understand the aim of those two different procedures. The ...
0
votes
0answers
16 views
Regression with pytorch with inferior results to ridge regression
I am trying to use a simple 3 layer neural net to predict a scaler output given an input of dimension 430. For my network, I use 2 layers of dimensions 600 and 80 and I use leakyReLU non-linearities. ...
0
votes
0answers
7 views
Matlab - Financial Modeling, Linear Regression with Prior
Am trying to implement this equation from the book Doing Data Science Straight Talk from the frontline, In chapter 6, page 161, equation below:
From what i can tell it is pretty much an enchanced ...
2
votes
1answer
52 views
Why is linear regression not a stable algorithm?
In the paper Stability and Generalization the author defines the stability of a learning algorithm, which intuitively means that changing one sample in the sample set does not affect the outcome much.
...
0
votes
0answers
15 views
Can we exclude non-significant variables (p>0.05) in ridge regression?
As far as I know, variables with p>0.05 are non-significant for the regression model.
I found similar questions here about p-value like:
What is the meaning of p values and t values in statistical ...
0
votes
0answers
57 views
Feature Selection for Ridge Regression
There is a closed-form equation for computing the optimal weight matrix $\mathbf{W}_{\text{optimal}}$ according to the feature matrix $\mathbf{\Phi}$ and the target matrix $\mathbf{Z}$ for ridge ...
1
vote
0answers
33 views
When to use LSTM vs Lasso/Ridge Regression vs ARIMA?
I have a set of N time series and want to make predictions about the future values of these N elementary time signals.
From a first rough analysis, I can say that at a given moment in time, the N ...
3
votes
2answers
59 views
glmnet for binary outcomes: Why is “%Dev” inversely correlated with lambda?
I am new to glmnet but would like to apply it to a dataset with binary outcomes. Can you please clarify a few questions for me? Below are the codes and data setup
<...
1
vote
0answers
14 views
Should Kernel Ridge Regression with linear kernel yield same results as Ridge regression?
I'm comparing the performance of different regressors from scikit-learn for fitting some data.
I would have expected that Ridge regression and Kernel Ridge regression both yield the same model/...
0
votes
0answers
14 views
What is the impact of the weight decay on self normalizing neural network with selu activations?
So there is a regularization technique called weight decay performing thikonov regularization (or in statistics community ridge regression).
There is also a (lets say new) approach for neural ...
0
votes
0answers
12 views
How should I consider the signs of the beta weights in a composite?
I had covariates (say, $X_1$) and some biomarkers ($X_2, \ldots, X_5$) and I wanted to model an outcome ($Y$) using these biomarkers. The biomarkers are correlated. So I decided to use a ridge ...
7
votes
2answers
195 views
Is there a canonical example of when ridge outperforms lasso?
Can someone please give me an example of when ridge would out perform lasso?
Won't lasso do better in most circumstances? If a regressor has a large coefficient, that means the regressor is a good ...
0
votes
0answers
16 views
Effect of log transformation or standardization of a regressor in the filtering step
We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
0
votes
1answer
25 views
What is the difference between “variables of interest” and “variables from which lasso selects” in Lasso?
I'm getting started with regularization models and I notice that Lasso requires three inputs, a dependent variable and then two sets of what I assume as independent variables, one which are of ...
2
votes
0answers
39 views
Getting started with regularization (Lasso)
I've got a small data set of 55 observations with a binary outcome variable of which only 11 are 1's and the rest are 0's.
I was wondering if Lasso was a useful tool to predict my outcome here and if ...
3
votes
0answers
26 views
How to consider interaction terms in the ridge / lasso / elastic net?
I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
1
vote
1answer
27 views
Comparison of regression models in terms of the importance of variables
I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values obtained are not on ...
15
votes
4answers
1k views
In regression, why not use regularization by default?
I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
2
votes
0answers
18 views
What are the available method that can alleviate the overfitting problem in traditional OLS problem, but still can get a linear fitting?
Recently, I have read the paper https://static1.squarespace.com/static/56def54a45bf21f27e160072/t/5a0d0673419202ef1b2259f2/1510803060244/The_Sampling_Error_in_Estimates_of_Mean-...
4
votes
1answer
91 views
AIC and its degrees of freedom for linear regression models
I have a dataset $S$ with $D$ features and three fitted linear regression models:
Model1. Ridge regression that is fitted on all $D$ features from $S$.
Model2. Ridge regression that is fitted on some $...
1
vote
1answer
34 views
False positive/negative rate in ridge and lasso regressions
I have a confusion matrix of true and estimated $\boldsymbol{\beta}$ vectors of lasso and ridge models from a replicate of a simulation study, say. The following tables illustrate the scenario.
$$\...
4
votes
0answers
51 views
Results of cv.glmnet in R versus RidgeCV in scikit-learn
I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R).
First a few things to note:
...
0
votes
0answers
12 views
Getting different values for MSE using anova(lm(y~.)) and mean(residuals(fit)^2)
Using this dataset of gas mileage for different cars I've been asked to run a ridge regression using $\frac{p*\sigma^2}{\beta'\beta}$ as the k-value.
I've been told
$\sigma^2 = MSE$
$p =$ ...
1
vote
1answer
31 views
Robust regression with M-estimators
I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator:
Is it possible/common to combine these with regularization terms, such ...
0
votes
0answers
26 views
Ridge Regression worse results with more feature. Does it make sense?
PREMISE
I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data.
I took care ...
1
vote
1answer
53 views
Poor performance on Regularized models
I'm trying to build a simple model to predict the price of a cab ride, using features
such as hour, source, destination, car model, distance, and
weather features such as pressure and humidity.
I've ...
1
vote
0answers
40 views
Efficiency of ridge regression in under determined systems
Imagine an underdetermined linear system, composed of N (continuous) labels and N samples, each have P features (with N < P):
$$\hat{\textbf{Y}}_{N \times 1} = \textbf{X}_{N \times P} \textbf{W}_{...
2
votes
1answer
109 views
Why cant Ridge Regression benift from negative lamda? [duplicate]
in Rigid regression, we generally set a positive Lambda for regularization to get a less Residual. Why cant we have a negative Lambda in a regularization if we can benefit from it?
0
votes
0answers
17 views
Estimating coefficients of a linear model with collinear dependent variables that have errors with known variances
I want to estimate the coefficients $\beta$ of the linear model
$Y=\beta X$
from observations of $(Y_i,X_i), i=1\ldots n$, where $X$ is multidimensional.
Two problems:
All variables have been ...
1
vote
1answer
67 views
Statistical library for orthogonal distance regression with a ridge penalty?
There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
0
votes
0answers
11 views
Variance of ridge regression estimator under perturbed data
In section 1.3 Ridge Regression as Perturbation of the notes the author comes up with the following ridge estimator
$$
\widehat{\boldsymbol{\beta}}=\left[(\mathbf{X}+\mathbf{W})^{T}(\mathbf{X}+\mathbf{...
0
votes
0answers
14 views
Regression training and testing error
Let’s say we fit a linear regression. What does the correlation between its training error and testing error say about the model, its performance or the data? What does a very low or very high ...
0
votes
1answer
37 views
l2 lambdas in Keras.regularizers [closed]
Is the value supplied to the shrinkage regularizers (l1 and l2) in Keras the inverse of the lambda coefficient?
e.g. ...
0
votes
0answers
13 views
How to do cross validation in ODE models with more predicted than measured time courses?
I have an ODE model of biochemical reactions with 37 state variables and 88 strictly positive parameters. Unfortunately, I can only expect to get time course measurements of about 10 state variables (...
0
votes
1answer
67 views
Derivation of ridge regression for multi-value-target vectors
At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$.
However, now I need to ...
0
votes
0answers
8 views
How To Feed Un-Scaled Data Into a Model That Was Trained on Scaled Data
I have a data frame that contains time series data. I split the dataframe into test and train. I want to prevent leakage so I split the data frame before doing any scaling.
On the train data set, I ...
3
votes
1answer
44 views
Why glmnet 's $\lambda$ value is so small? Does it strictly implement the loss function under the hood?
I am running a glmnet fit with 1200000 samples.
According to the glmnet doc, $\lambda$ value is the coefficient controlling how much the regularization term contributes to the total loss function.
...
0
votes
0answers
23 views
Estimate variance of parameters in non-linear ridge regression
I am basically estimating the shape of an unknown function f(x) from multi-dimensional chemical reaction data by estimating the most likely function values $f$ on a grid $x$ with kernel regression. To ...
0
votes
0answers
43 views
Solving ridge regression for p >> n case using dual algorithm with or without nonnegativity constraints
I was reading the paper
"Efficient Regularized Regression with
L0 Penalty for Variable Selection and Network Construction"
in which iterated ridge regression is used to solve L0 penalized regression ...
1
vote
1answer
34 views
High odds ratio for composite score created by ridge regression
This question is a follow up to one of my previous questions asked on this site. The goal was to create a composite score for biomarkers related to a binary outcome and then use that in a regression ...
0
votes
0answers
43 views
Comparing Ridge and Lasso Regression [duplicate]
I was thinking about main differences between ridge and lasso introducing a $\ell^2$ and $\ell^1$ penalty term respectively. The main thing is that with ridge I will keep all my features in the end ...
2
votes
0answers
100 views
Shapley value vs ridge regression
My goal is to get the feature importance for multiple regression. I have a data set with some multicollinearity. I found two methods to solve this problem.
The first one is the Shapley value. ...
2
votes
1answer
215 views
L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)
I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve.
I have successfully run both glmnet() and cv....
0
votes
0answers
85 views
Mathematical proof of how L1 and L2 regularization work [duplicate]
How do you mathematically prove that L1 regularization makes weights sparse but L2 regularization does not?
1
vote
2answers
508 views
Implementing ridge regression in python
I was trying to implement ridge regression in python. I implemented the following code:
...
0
votes
0answers
10 views
What is the meaning of assuming a special prior on regularization method
I have heard/read that L1 regularization assumes Laplacian prior, however L2 regularization assumes Gaussian prior.
But what exactly "assume" mean here? How does it work? How do each of these ...
1
vote
0answers
7 views
Does the initial distribution of data have any affect on which regularization parameter can work well?
In scenarios when we want to know why performance of a predicting linear regression model when using L1 regularization has outperformed with the case that we have used L2 regularization, I wonder ...
0
votes
0answers
63 views
Ridge regression is similar to Linear regression [duplicate]
I can not see any difference between
Ridge Regression and Linear Regression
MY understanding, The point of ridge Regression is based on the training data we find the best line that fits training ...
0
votes
1answer
58 views
Understanding concepts of regularization
I am trying to understand regularization in machine learning. But, I do not understand some fundamental concepts in this topic, could you please explain?
A model that has high variance, captures ...
