# Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

685 questions
496 views

### Different regularization parameter per parameter

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the ...
39k views

### How to interpret glmnet?

I am trying to fit a multivariate linear regression model with approximately 60 predictor variables and 30 observations, so I am using the glmnet package for regularized regression because p>n. I ...
301 views

### Standardizing response variable in shrinkage/regularization

I know that I should standardize my predictors before estimating something like Lasso. But what about the response variable? Do I standardise this as well? Only ...
18k views

### Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
631 views

### When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
311 views

### Lasso ||a|| and “General Lasso” ||Da||

Ryan Tibshirani introduced once a more general type of Lasso, where the regularizer is $$\parallel D \alpha \parallel_1$$ instead of $\parallel \alpha \parallel_1$. See paper However, there is nearly ...
107 views

### Kernel/Basis function design with regularizer

I am solving this problem: $$\sum_i \parallel f(x_i)- y_i\parallel_2^2 + \lambda <\psi f, \psi f>_{L_2}^2$$ where the second part $<\psi f, \psi f>_2^2$ is regularizer using the linear ...
178 views

### Running regularized logistic regressions on very large datasets

I want to run a regularized logistic regression on a dataset with 25 million observations and about a 1000 mostly non-sparse columns with non-ignorable weights. My first choice would be BayesGLM, ...
18k views

### Feature selection & model with glmnet on Methylation data (p>>N)

I would like to use GLM and Elastic Net to select those relevant features + build a linear regression model (i.e., both prediction and understanding, so it would be better to be left with relatively ...
33k views

### Neural Networks: weight change momentum and weight decay

Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations: $$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t),$$ ...
262 views

### How big are regularization parameters values?

I wanted to know how big are the regularization parameter values for ridge or lasso. I have seen most of the places generally using values like 0.1 or 0.01 but in some of my experiments the cross ...
35k views

### How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term: $$\beta = (X^TX)^{-1}X^Ty.$$ But after adding the ...
5k views

### What does it mean if all the coefficient estimates in a lasso regression converge to zero?

I attempted to run lasso on a 12 X 52 matrix (11 predictors) using this MATLAB function http://www.mathworks.com.au/help/stats/lasso.html. I found that the results converged to zero. How should I ...
63 views

### Is it bad to leave in poor predictors in regularized multiple regression?

There are some variables that I measured but strongly suspect are useless because (for example) almost all my data points scored the same on that (binary) variable. It's been put to me that I may ...
139 views

### Needle-in-a-haystack Regularized Regression

I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the ...
90 views

### Maximum risk and sparse estimation

On Larry Wasserman's blog, he talks about the "Steep price of sparsity" here: http://normaldeviate.wordpress.com/2013/07/27/the-steep-price-of-sparsity/ In it, he points out that a sparse estimation ...
5k views

### Regularization and feature scaling in online learning?

Let's say I have a logistic regression classifier. In normal batch learning, I'd have a regularizer term to prevent overfitting and keep my weights small. I'd also normalize and scale my features. In ...
13k views

### (Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
2k views

### Why does regularization of coefficient magnitude improve the generalization of linear regression? [duplicate]

What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that In many cases, tikhonov matrix is chosen as ...
348 views

### Is regularization required with overdetermined data

I'm doing least squares estimation on large set of data and I started to wonder whether I should regularize my OLS estimator. My professor told me that this isn't necessary, because the data is ...
89 views

### Sparsity regularization for eigenvectors

One way to think about finding the eigenvectors of a matrix $A$ is that they are the critical points of the functional $\vec x^\top A \vec x$ subject to $\|\vec x\|_2=1$. To regularize this problem, ...
3k views

### Effect of features that are highly correlated with each other on a decision tree

I have a dataset of roughly 500 features and am training a binary classifier using GBM - gradient boosted machines, an ensemble of decision trees. Of these 500 variables, I am sure some are highly ...
8k views

### Coefficients paths – comparison of ridge, lasso and elastic net regression

I would like to compare models selected with ridge, lasso and elastic net. Fig. below shows coefficients paths using all 3 methods: ridge (Fig A, alpha=0), lasso (Fig B; alpha=1) and elastic net (Fig ...
704 views

### GBDT and model building: How am I overfitting?

Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT ...
75 views

### Confusion related to regularization parameter selection by cross validation

I can see lots of paper mentioning they selected some parameters like regularization parameter $\lambda$ by cross validation. What do they mean by that?
366 views

### Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
6k views

### What does “degree of freedom” mean in neural networks?

In Bishop's book "Pattern Classification and Machine Learning", it describes a technique for regularization in the context of neural networks. However, I don't understand a paragraph describing that ...
217 views

### Forward Stepwise selection

I am assuming the following model: $Y = \beta X + \epsilon$ Here both $X$ and $Y$ are matrices. I fit the least squares model without any regularization and get the matrix $\beta$. I would like to ...
2k views

### Alternatives to glmnet for feature selection on data with lots of NAs

I have a surgical database in which there are approximately 100,000 observations and 200 features. Each observation corresponds to a separate patient while the features correspond to either ...
86 views

### Question on the usage of regularization in applied statistics/science

I was reading the paper A significance test for the lasso'' by Lockhart, Tibshirani et al and was considering the issue of applying regularization in the applied sciences (for example, behavioral ...
473 views

### model selection with glmnet

I am trying to fit a multinomial logit model using glmnet. I have a few questions: How is the baseline category specified? Looking at the model coefficients using coef.glmnet, I'm thinking that many ...
538 views

### Classification with 3 groups, repeated measurements, missing values, more predictors than subjects

I am working on a classification problem with the following characteristics: Individuals belong to one of three groups. The groups are "somewhat ordinal": controls, subclinical and clinical group. ...
179 views

36k views

### Why L1 norm for sparse models

I am reading the books about linear regression. There are some sentences about the L1 and L2 norm. I know them, just don't understand why L1 norm for sparse models. Can someone use give a simple ...
148 views

### Enforcing sparsity on probability [closed]

I am trying to induce a probability distribution $Q$ by optimizing an objective function and am wondering how can one encourage sparsity for $Q$ while keeping the optimization convex. In particular, ...
649 views

### How to obtain good performance (low error rate) on massive data set?

Suppose I have massive data set (think Terabytes) is available to train a learning algorithm. Which one of the following conditions must be true to obtain good performance (low error rate) a. Using ...
398 views

### Robust regularized regression

I've been using elastic net implemented in R (via glmnet) for some modeling, but I was wondering, due to the number of outliers in my data, if there was some sort of modeling approach for regularized ...
2k views

### Advantages of doing “double lasso” or performing lasso twice?

I once heard a method of using the lasso twice (like a double-lasso) where you perform lasso on the original set of variables, say S1, obtain a sparse set called S2, and then perform lasso again on ...
17k views

### libsvm “reaching max number of iterations” warning and cross-validation [closed]

I'm using libsvm in C-SVC mode with a polynomial kernel of degree 2 and I'm required to train multiple SVMs. Each training set has 10 features and 5000 vectors. During training, I am getting this ...