# Tagged Questions

A regularization method for regression models that shrinks coefficients towards zero.

21 views

### Lasso and Ridge tuning parameter scope

In ridge and lasso linear regression, an important step is to choose the tuning parameter lambda, often I use grid search on log scale from -6->4, it works well on ridge, but on lasso, should I take ...
8 views

### fast way to train a classifier on different but overlapping features

I am training a linear classifier repeatedly on different set of overlapping features. I have a 3D grid of features, each time features from a small sphere from a grid are used to train a classifier, ...
42 views

### Can L1 linear regression perform worse than vanilla linear regression on fewer features?

I have a data set with 2 features and I'm trying to predict one real-valued variable. I use linear regression and I measure the error using 10-fold CV and absolute mean error as a metric. I noticed ...
9 views

### Generalized Tikhonov regularization in glmnet?

Is it possible to do Generalized Tikhonov regularization https://en.wikipedia.org/wiki/Tikhonov_regularization#Generalized_Tikhonov_regularization with glmnet? This seems like straightforward and ...
31 views

### Defining lambda value for ridge regression

I try to understand ridge regression. I think the most important point in it is defining lambda value. I researched some R codes but I didn't understand how to define it. In the example here, in the ...
40 views

### doubt on ridge regression [duplicate]

Assume you have a training dataset consisting of 1 million observations. Suppose running the closed-form solution to fit a multiple linear regression model using ridge regression on this data takes 1 ...
30 views

### Interpreting Special Case for Ridge Regression and the Lasso

The below text is from Statistical Learning Page no.225 Consider a case with $n = p$, and $\mathbf{X}$ a diagonal matrix with 1’s on the diagonal and 0’s in all oﬀ-diagonal elements. To simplify ...
25 views

### Tikhonov regularization in the context of deconvolution

I came across "Tikhonov regularization" and I have bare knowledge on it. It seems that it is a type of regularization that is important for deconvolution. Are there any good resources and examples? ...
27 views

### Predicting of revenue. Penalized regression (Ridge regression)

I have data of sales. I've selected one point of sales to check a possibility of predicting revenue using regression method (I don't know what can I use in this task). First of all I've tried to find ...
72 views

### double feature value in ridge regression, coefficients change?

In ridge regression using unnormalized features, if you double the value of a given feature A (i.e., a specific column of the feature matrix), what happens to the estimated coefficients for every ...
22 views

### Replacing ridge regression with Bayesian MCMC

I have a ridge regression model $y = \beta_1 x_1 + \beta_2 x_2 + ...$ The $x$s are highly collinear but are all physically relevant, hence use of ridge regression. And am considering replacing ...
58 views

### Can I implement ridge regression in terms of OLS regression?

Can I implement ridge regression in terms of OLS regression? Is it even possible? I am interested because scikit-learn supports non-negative least squares (NNLS), ...
64 views

### How to perform non-negative ridge regression?

How to perform non-negative ridge regression? Non-negative lasso is available in scikit-learn, but for ridge, I cannot enforce non-negativity of betas, and indeed, ...
19 views

### Difference of feature importance from Random Forest and Regularized Logistic Regression

I have 13 features in a classification task and I use Random Forest, L1 logistic regression and L2 logistic regression for as separate classifiers and would like to compare their performance. Although ...
5 views

### Bounding the loss for kernel regularization algorithms

Some learning bounds depend on a quantity $M$ that is the maximum loss a learning algorithm can have: so $L(h(x),y) \leq M$, where $h(x)$ is the prediction of the model, and $y$ is the label, and $L$ ...
9 views

22 views

### Why is intercept excluded in ridge regression? [duplicate]

Why don't we include the intercept term in ridge regression, i.e exclude $w_0$ term in calculating the coefficients? Why is it so?
40 views

### The scaling problem of ridge regression

I have been confused with the scaling of ridge regression input for a long time. There are several sources about how to do the scaling: Just do the centering to input(From "The Elements of ...
58 views

### Decompose ridge regression bias error into model bias and estimation bias

How can I show that the in-sample bias error in Ridge regression can be decomposed into model bias plus estimation bias? I.e., if $Avg$ takes the average over all the input variables $x$ in the ...
157 views

### Computing cross-validated $R^2$ from mean cross-validation error

I am currently using cv.glmnet in R. I would like to compute both a training $R^2$ and a cross-validated $R^2$. R gives mean cross-validated error and for the ...
48 views

### Meaning and significance testing of coefficients in lasso/ridge regression

Can somebody explain the importance of significance testing in ridge/lasso regression? Is it necessary to do it? And, how can we interpret the coefficients of ridge regression which are penalized?
90 views

### Elastic net produces complex output with too many non-zero coefficients

I have run 3-fold cross-validation for elastic net using elasticnet R function on ~200 observations and using 80 variables (and there will be some more). Both ...
298 views

### Why is ridge regression giving different results in Matlab and Python?

Why is the output from Matlab and Python vary for ridge regression? I use the ridge command in Matlab and scikit-learn in Python ...
116 views

### Can ridge regression be used in the presence of categorical predictors?

I have a regression problem and I am thinking of using ridge regression. One of the predictors is subject's gender, which is a categorical variable. How to take care of this variable for ridge ...
51 views

### First step of Kernel Ridge Regression?

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
10 views

### First step of Kernal Ridge Regression? [duplicate]

I want to implement kernel ridge regression (KRR) using a polynomial kernel as a function that takes the training objects, training labels and test objects as arguments, and outputs the vector of ...
66 views

### Do methods exist other than Ridge Regression and Y ~ X + 0 to prevent OLS from dropping variables?

Goal is to evaluate chess players using a novel analysis system I'm been working on -- not all wins are created equal, finding the only move in razor sharp positions is better than finding the best ...
523 views

### What is elastic net regularization, and how does it solve the drawbacks of Ridge (L2) and Lasso (L1)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
586 views

### When will L1 regularization work better than L2 and vice versa?

Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant. How to decide which regularization (L1 or L2) to ...
14 views

### How to model the discrepancy between test result and prediction code, and find the main sources of discrepancy

I have data about some tested machines, stored as a database with $\approx$ 300 observations, 40 predictors, 2 outcomes. The two outcomes (responses) are the result of the machine test for that ...
15 views

### Ridge regression on centred or uncentered data? [duplicate]

Today is the first day that I have heard about ridge regression, so please do not judge harshly. I have learned that you use ridge regression when there is a case for multicollinearity of your ...
161 views

### Why is using centered or uncentered data equivalent in ridge regression?

Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} ...
199 views

### Implementing kernel ridge regression

I want to implement kernel ridge regression in R. My problem is that I can't figure out how to generate the kernel values and I do not know how to use them for the ridge regression. Before going to ...
108 views

### Recommend a method for variable selection (other than classification tree or random forest)?

Just wonder if you could recommend a few methods (other than tree-based methods) to analyze a dataset in which n= 350 and p = 35. The goal is not so much about prediction, but to find/select ...
2k views

### Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
340 views

### Why can't ridge regression provide better interpretability than LASSO?

I already have an idea about pros and cons of ridge regression and the LASSO. For the LASSO, L1 penalty term will yield a sparse coefficient vector, which can be viewed as a feature selection method. ...
I'm trying to learn some basic Machine Learning and some basic R. I have made a very naive implementation of $L_2$ regularization in R based on the formula: \$\hat w^{ridge} = (X^TX +\lambda I)^{-1} ...