A regularization method for regression models that shrinks coefficients towards zero.

learn more… | top users | synonyms (1)

0
votes
0answers
11 views

applying Posterior predictive distribution on the data from which the coefficient of regression were estimated [on hold]

I am new to linear regression modelling. For the given linear model $Y_{current} = \mu+ \beta X_{current} + \varepsilon \sim N(0|\sigma^2)$ Generally, in regression analysis we estimate, coefficient ...
1
vote
0answers
68 views

How freqently are the information conditons for proper Akaike information criterion application actually met?

Akaike information criterion (AIC) is limited to goodness-of-fit for assumed distributions. That is, for an assumed distribution (especially homoscedastic) one can assume a maximum likelihood ...
3
votes
1answer
192 views

Ridge/Lasso Lambda greater than 1

I ran Ridge and Lasso regressions using an algorithm to automatically find the optimum lambda. However, the algorithm couldn't find an optimum lambda between 0 and 1. In some cases I could find ...
1
vote
0answers
29 views

Understanding Kernel Ridge Regression and How It Works (and Implementing it in R)

I am trying to understand how KRR works for drug-protein-interaction and many aspects of it seem very confusing. Supposing I have a data set as follows of Drug-Protein interactions; values show how ...
3
votes
1answer
98 views

How is L1 regulaziation derived?

I understand the basic idea of regularization. I am very curious to know the derivations behind it so that I get the complete picture. I though a good place to start leraning about regularization is ...
1
vote
0answers
12 views

OLS or Ridge in Multicollinearity data

I am new to stats and linear regression. I just want to understand the exact scenario and usage between Ridge and OLS. Here is the data sample i have been using. In this both Weight and BSA are ...
2
votes
2answers
55 views

Lasso and Ridge tuning parameter scope

In ridge and lasso linear regression, an important step is to choose the tuning parameter lambda, often I use grid search on log scale from -6->4, it works well on ridge, but on lasso, should I take ...
0
votes
0answers
12 views

fast way to train a classifier on different but overlapping features

I am training a linear classifier repeatedly on different set of overlapping features. I have a 3D grid of features, each time features from a small sphere from a grid are used to train a classifier, ...
0
votes
0answers
46 views

Can L1 linear regression perform worse than vanilla linear regression on fewer features?

I have a data set with 2 features and I'm trying to predict one real-valued variable. I use linear regression and I measure the error using 10-fold CV and absolute mean error as a metric. I noticed ...
1
vote
0answers
15 views

Generalized Tikhonov regularization in glmnet?

Is it possible to do Generalized Tikhonov regularization https://en.wikipedia.org/wiki/Tikhonov_regularization#Generalized_Tikhonov_regularization with glmnet? This seems like straightforward and ...
0
votes
1answer
36 views

Defining lambda value for ridge regression

I try to understand ridge regression. I think the most important point in it is defining lambda value. I researched some R codes but I didn't understand how to define it. In the example here, in the ...
3
votes
1answer
40 views

Interpreting Special Case for Ridge Regression and the Lasso

The below text is from Statistical Learning Page no.225 Consider a case with $n = p$, and $\mathbf{X}$ a diagonal matrix with 1’s on the diagonal and 0’s in all off-diagonal elements. To simplify ...
0
votes
1answer
28 views

Tikhonov regularization in the context of deconvolution

I came across "Tikhonov regularization" and I have bare knowledge on it. It seems that it is a type of regularization that is important for deconvolution. Are there any good resources and examples? ...
0
votes
0answers
37 views

Predicting of revenue. Penalized regression (Ridge regression)

I have data of sales. I've selected one point of sales to check a possibility of predicting revenue using regression method (I don't know what can I use in this task). First of all I've tried to find ...
2
votes
0answers
104 views

double feature value in ridge regression, coefficients change?

In ridge regression using unnormalized features, if you double the value of a given feature A (i.e., a specific column of the feature matrix), what happens to the estimated coefficients for every ...
0
votes
0answers
24 views

Replacing ridge regression with Bayesian MCMC

I have a ridge regression model $ y = \beta_1 x_1 + \beta_2 x_2 + ...$ The $x$s are highly collinear but are all physically relevant, hence use of ridge regression. And am considering replacing ...
1
vote
0answers
59 views

Can I implement ridge regression in terms of OLS regression?

Can I implement ridge regression in terms of OLS regression? Is it even possible? I am interested because scikit-learn supports non-negative least squares (NNLS), ...
4
votes
1answer
73 views

How to perform non-negative ridge regression?

How to perform non-negative ridge regression? Non-negative lasso is available in scikit-learn, but for ridge, I cannot enforce non-negativity of betas, and indeed, ...
0
votes
0answers
20 views

Difference of feature importance from Random Forest and Regularized Logistic Regression

I have 13 features in a classification task and I use Random Forest, L1 logistic regression and L2 logistic regression for as separate classifiers and would like to compare their performance. Although ...
1
vote
0answers
5 views

Bounding the loss for kernel regularization algorithms

Some learning bounds depend on a quantity $M$ that is the maximum loss a learning algorithm can have: so $L(h(x),y) \leq M$, where $h(x)$ is the prediction of the model, and $y$ is the label, and $L$ ...
0
votes
0answers
9 views

help me understand the proof in the paper “restricted ridge estimation”

I'm reading the paper "restricted ridge estimation" by Grob(2003). I can not understand the proof of theorem 1 in this paper. I don't know how this estimator $\hat{\beta}_{r}(k) = ...
6
votes
1answer
430 views

Is regression with L1 regularization the same as Lasso, and with L2 regularization the same as ridge regression? And how to write “Lasso”?

I'm a software engineer learning machine learning, particularly through Andrew Ng's machine learning courses. While studying linear regression with regularization, I've found terms that are confusing: ...
5
votes
1answer
88 views

Lucid explanation for “numerical stability of matrix inversion” in ridge regression and its role in reducing overfit

I understand that we can employ regularization in a least squares regression problem as $$\boldsymbol{w}^* = \operatorname*{argmin}_w \left[ (\mathbf y-\mathbf{Xw})^T(\boldsymbol{y}-\mathbf{Xw}) + ...
0
votes
1answer
25 views

Best subset algorithm for ridge regression in R [closed]

I'm searching for a best subset selection algorithm for ridge regression in R. There is a wide range of algoritms for an ordinary least squares fit. There also exists a function like ...
1
vote
1answer
57 views

What happens if you square an RBF kernel function?

Let's say we use a kernel regularization algorithm such as ridge regression to minimize some loss in an RBF kernel: $$\min_{h \in H} \frac{1}{n} \sum_i (h(x_i) - y(x_i))^2 + ||h||^2_K$$ We get some ...
0
votes
0answers
17 views

Cross-validation results differ from package

I am trying to test out a hand-rolled cross-validation procedure for ridge regression. I've run the glmnet package which gives me an MSE bottoming out around ...
1
vote
1answer
66 views

Deriving the Ridge Regression $\boldsymbol{\beta}\mid \mathbf{y}$ distribution

Apparently the estimate $\hat{\boldsymbol{\beta}}$ for ridge regression comes up as the mean or mode of the posterior distribution given by $f_{\boldsymbol{\beta}\mid \mathbf{y}}$. This is the ...
1
vote
3answers
86 views

Why are solution to ridge regression always expressed using matrix notation?

Consider the following ridge regression problem: minimize the loss function $\sum_{i=1}^n ||y_i - w^T x_i||_2^2 + \lambda ||w||_2^2$ with respect to the weight vector w. Taking derivative with respect ...
3
votes
1answer
73 views

Ridge regression formulation as constrained versus penalized: How are they equivalent?

I seem to be misunderstanding a claim about linear regression methods that I've seen in various places. The parameters of the problem are: Input: $N$ data samples of $p+1$ quantities each ...
3
votes
2answers
157 views

Ridge regression — why does the model only care to control large outliers?

One of the purposes of ridge regression is to curb the effects of outliers which may cause the regression coefficients to be so large and hence cause a highly biased model. That's why the constraint ...
1
vote
0answers
36 views

Ridge Regression Centering Proof [duplicate]

This is a ridge regression problem. The following two problems are equivalent: $(w_t, b_\lambda ) = argmin_{w,b}\{\sum_{i=1}^m (y_i-b-w^Tx_i)^2+\lambda w^Tw\} $ $(w_t, b_\lambda )= ...
7
votes
3answers
272 views

How to find regression coefficients $\beta$ in ridge regression?

In ridge regression, the objective function to be minimized is: $$\text{RSS}+\lambda \sum\beta_j^2.$$ Can this be optimized using the Lagrange multiplier method? Or is it straight differentiation?
1
vote
0answers
84 views

Alternatives to Box-Tidwell transformation for ridge regression?

I would like to fit the following model by ridge regression (the xs correlate strongly with one another) $y = \beta_1 {x_1}^{\lambda_1} + \beta_2 {x_2}^{\lambda_2} + \beta_3 {x_3}^{\lambda_3} + ...
0
votes
0answers
22 views

Why is intercept excluded in ridge regression? [duplicate]

Why don't we include the intercept term in ridge regression, i.e exclude $w_0$ term in calculating the coefficients? Why is it so?
0
votes
0answers
42 views

The scaling problem of ridge regression

I have been confused with the scaling of ridge regression input for a long time. There are several sources about how to do the scaling: Just do the centering to input(From "The Elements of ...
2
votes
1answer
60 views

Decompose ridge regression bias error into model bias and estimation bias

How can I show that the in-sample bias error in Ridge regression can be decomposed into model bias plus estimation bias? I.e., if $Avg$ takes the average over all the input variables $x$ in the ...
6
votes
1answer
162 views

Computing cross-validated $R^2$ from mean cross-validation error

I am currently using cv.glmnet in R. I would like to compute both a training $R^2$ and a cross-validated $R^2$. R gives mean cross-validated error and for the ...
1
vote
0answers
52 views

Meaning and significance testing of coefficients in lasso/ridge regression

Can somebody explain the importance of significance testing in ridge/lasso regression? Is it necessary to do it? And, how can we interpret the coefficients of ridge regression which are penalized?
2
votes
1answer
96 views

Elastic net produces complex output with too many non-zero coefficients

I have run 3-fold cross-validation for elastic net using elasticnet R function on ~200 observations and using 80 variables (and there will be some more). Both ...
3
votes
1answer
322 views

Why is ridge regression giving different results in Matlab and Python?

Why is the output from Matlab and Python vary for ridge regression? I use the ridge command in Matlab and scikit-learn in Python ...
3
votes
2answers
151 views

Can ridge regression be used in the presence of categorical predictors?

I have a regression problem and I am thinking of using ridge regression. One of the predictors is subject's gender, which is a categorical variable. How to take care of this variable for ridge ...
1
vote
0answers
52 views

First step of Kernel Ridge Regression?

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
0
votes
0answers
10 views

First step of Kernal Ridge Regression? [duplicate]

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
0
votes
1answer
66 views

Do methods exist other than Ridge Regression and Y ~ X + 0 to prevent OLS from dropping variables?

Goal is to evaluate chess players using a novel analysis system I'm been working on -- not all wins are created equal, finding the only move in razor sharp positions is better than finding the best ...
8
votes
2answers
584 views

What is elastic net regularization, and how does it solve the drawbacks of Ridge (L2) and Lasso (L1)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
12
votes
2answers
606 views

When will L1 regularization work better than L2 and vice versa?

Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant. How to decide which regularization (L1 or L2) to ...
2
votes
0answers
14 views

How to model the discrepancy between test result and prediction code, and find the main sources of discrepancy

I have data about some tested machines, stored as a database with $\approx$ 300 observations, 40 predictors, 2 outcomes. The two outcomes (responses) are the result of the machine test for that ...
0
votes
0answers
15 views

Ridge regression on centred or uncentered data? [duplicate]

Today is the first day that I have heard about ridge regression, so please do not judge harshly. I have learned that you use ridge regression when there is a case for multicollinearity of your ...
2
votes
1answer
167 views

Why is using centered or uncentered data equivalent in ridge regression?

Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: \begin{equation} (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} ...
1
vote
0answers
245 views

Implementing kernel ridge regression

I want to implement kernel ridge regression in R. My problem is that I can't figure out how to generate the kernel values and I do not know how to use them for the ridge regression. Before going to ...