Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

2
votes
3answers
496 views

Different regularization parameter per parameter

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the ...
34
votes
2answers
39k views

How to interpret glmnet?

I am trying to fit a multivariate linear regression model with approximately 60 predictor variables and 30 observations, so I am using the glmnet package for regularized regression because p>n. I ...
2
votes
0answers
301 views

Standardizing response variable in shrinkage/regularization

I know that I should standardize my predictors before estimating something like Lasso. But what about the response variable? Do I standardise this as well? Only ...
68
votes
3answers
18k views

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
5
votes
0answers
631 views

When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
4
votes
1answer
311 views

Lasso ||a|| and “General Lasso” ||Da||

Ryan Tibshirani introduced once a more general type of Lasso, where the regularizer is $$\parallel D \alpha \parallel_1$$ instead of $\parallel \alpha \parallel_1$. See paper However, there is nearly ...
0
votes
0answers
107 views

Kernel/Basis function design with regularizer

I am solving this problem: $$ \sum_i \parallel f(x_i)- y_i\parallel_2^2 + \lambda <\psi f, \psi f>_{L_2}^2 $$ where the second part $<\psi f, \psi f>_2^2$ is regularizer using the linear ...
4
votes
1answer
178 views

Running regularized logistic regressions on very large datasets

I want to run a regularized logistic regression on a dataset with 25 million observations and about a 1000 mostly non-sparse columns with non-ignorable weights. My first choice would be BayesGLM, ...
28
votes
1answer
18k views

Feature selection & model with glmnet on Methylation data (p>>N)

I would like to use GLM and Elastic Net to select those relevant features + build a linear regression model (i.e., both prediction and understanding, so it would be better to be left with relatively ...
40
votes
1answer
33k views

Neural Networks: weight change momentum and weight decay

Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations: $$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t),$$ ...
1
vote
1answer
262 views

How big are regularization parameters values?

I wanted to know how big are the regularization parameter values for ridge or lasso. I have seen most of the places generally using values like 0.1 or 0.01 but in some of my experiments the cross ...
33
votes
5answers
35k views

How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term: $$\beta = (X^TX)^{-1}X^Ty.$$ But after adding the ...
4
votes
1answer
5k views

What does it mean if all the coefficient estimates in a lasso regression converge to zero?

I attempted to run lasso on a 12 X 52 matrix (11 predictors) using this MATLAB function http://www.mathworks.com.au/help/stats/lasso.html. I found that the results converged to zero. How should I ...
1
vote
1answer
63 views

Is it bad to leave in poor predictors in regularized multiple regression?

There are some variables that I measured but strongly suspect are useless because (for example) almost all my data points scored the same on that (binary) variable. It's been put to me that I may ...
4
votes
1answer
139 views

Needle-in-a-haystack Regularized Regression

I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the ...
3
votes
0answers
90 views

Maximum risk and sparse estimation

On Larry Wasserman's blog, he talks about the "Steep price of sparsity" here: http://normaldeviate.wordpress.com/2013/07/27/the-steep-price-of-sparsity/ In it, he points out that a sparse estimation ...
12
votes
3answers
5k views

Regularization and feature scaling in online learning?

Let's say I have a logistic regression classifier. In normal batch learning, I'd have a regularizer term to prevent overfitting and keep my weights small. I'd also normalize and scale my features. In ...
32
votes
3answers
13k views

(Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
7
votes
3answers
2k views

Why does regularization of coefficient magnitude improve the generalization of linear regression? [duplicate]

What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that In many cases, tikhonov matrix is chosen as ...
0
votes
1answer
348 views

Is regularization required with overdetermined data

I'm doing least squares estimation on large set of data and I started to wonder whether I should regularize my OLS estimator. My professor told me that this isn't necessary, because the data is ...
3
votes
0answers
89 views

Sparsity regularization for eigenvectors

One way to think about finding the eigenvectors of a matrix $A$ is that they are the critical points of the functional $\vec x^\top A \vec x$ subject to $\|\vec x\|_2=1$. To regularize this problem, ...
0
votes
1answer
3k views

Effect of features that are highly correlated with each other on a decision tree

I have a dataset of roughly 500 features and am training a binary classifier using GBM - gradient boosted machines, an ensemble of decision trees. Of these 500 variables, I am sure some are highly ...
11
votes
1answer
8k views

Coefficients paths – comparison of ridge, lasso and elastic net regression

I would like to compare models selected with ridge, lasso and elastic net. Fig. below shows coefficients paths using all 3 methods: ridge (Fig A, alpha=0), lasso (Fig B; alpha=1) and elastic net (Fig ...
1
vote
0answers
704 views

GBDT and model building: How am I overfitting?

Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT ...
0
votes
1answer
75 views

Confusion related to regularization parameter selection by cross validation

I can see lots of paper mentioning they selected some parameters like regularization parameter $\lambda$ by cross validation. What do they mean by that?
10
votes
0answers
366 views

Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
13
votes
4answers
6k views

What does “degree of freedom” mean in neural networks?

In Bishop's book "Pattern Classification and Machine Learning", it describes a technique for regularization in the context of neural networks. However, I don't understand a paragraph describing that ...
1
vote
0answers
217 views

Forward Stepwise selection

I am assuming the following model: $Y = \beta X + \epsilon$ Here both $X$ and $Y$ are matrices. I fit the least squares model without any regularization and get the matrix $\beta$. I would like to ...
3
votes
3answers
2k views

Alternatives to glmnet for feature selection on data with lots of NAs

I have a surgical database in which there are approximately 100,000 observations and 200 features. Each observation corresponds to a separate patient while the features correspond to either ...
3
votes
1answer
86 views

Question on the usage of regularization in applied statistics/science

I was reading the paper ``A significance test for the lasso'' by Lockhart, Tibshirani et al and was considering the issue of applying regularization in the applied sciences (for example, behavioral ...
2
votes
1answer
473 views

model selection with glmnet

I am trying to fit a multinomial logit model using glmnet. I have a few questions: How is the baseline category specified? Looking at the model coefficients using coef.glmnet, I'm thinking that many ...
4
votes
1answer
538 views

Classification with 3 groups, repeated measurements, missing values, more predictors than subjects

I am working on a classification problem with the following characteristics: Individuals belong to one of three groups. The groups are "somewhat ordinal": controls, subclinical and clinical group. ...
2
votes
1answer
179 views

Parameter estimate for linear regression with regularization

For given cost function $S(\beta) = (Y - X \beta)^T(Y - X \beta) + \lambda \beta^T \beta$, where $\lambda$ is regularization parameter, the $\beta$ that minimizes the given cost function is $\beta = [...
5
votes
1answer
735 views

Can the bias introduced by lasso change the sign of a coefficient?

L1 penalized regression introduces a bias on your regression model but decreases the variance. When this bias is introduced, is it possible that the coefficient of $B$ changes sign? This would ...
0
votes
1answer
104 views

Reducing the dimensionality of a problem

My particular application needs me to build a linear model with a strong correlation structure amongst the independent variables. The dimensions of the problem are high, for instance 1million X 200. ...
2
votes
0answers
227 views

kernelized l1 norm and the representer theorem

I'm trying to derive a kernel-ized $l_1$ penalty for logistic regression. I have been looking at the slides Learning with Sparsity Inducing Norms along with the slides on Regularization and Variable ...
5
votes
1answer
755 views

SVM optimization problem

I think I understand the main idea in support vector machines. Let us assume that we have two linear separable classes and want to apply SVMs. What SVM is doing is that it searches a hyperplane $\{\...
85
votes
6answers
36k views

Why L1 norm for sparse models

I am reading the books about linear regression. There are some sentences about the L1 and L2 norm. I know them, just don't understand why L1 norm for sparse models. Can someone use give a simple ...
1
vote
0answers
148 views

Enforcing sparsity on probability [closed]

I am trying to induce a probability distribution $Q$ by optimizing an objective function and am wondering how can one encourage sparsity for $Q$ while keeping the optimization convex. In particular, ...
0
votes
1answer
649 views

How to obtain good performance (low error rate) on massive data set?

Suppose I have massive data set (think Terabytes) is available to train a learning algorithm. Which one of the following conditions must be true to obtain good performance (low error rate) a. Using ...
3
votes
2answers
398 views

Robust regularized regression

I've been using elastic net implemented in R (via glmnet) for some modeling, but I was wondering, due to the number of outliers in my data, if there was some sort of modeling approach for regularized ...
25
votes
2answers
2k views

Advantages of doing “double lasso” or performing lasso twice?

I once heard a method of using the lasso twice (like a double-lasso) where you perform lasso on the original set of variables, say S1, obtain a sparse set called S2, and then perform lasso again on ...
19
votes
1answer
17k views

libsvm “reaching max number of iterations” warning and cross-validation [closed]

I'm using libsvm in C-SVC mode with a polynomial kernel of degree 2 and I'm required to train multiple SVMs. Each training set has 10 features and 5000 vectors. During training, I am getting this ...
2
votes
1answer
2k views

matlab gmdistribution.fit 'Regularize' - what regularization method?

I am wondering what is behind matlab 'Regularize' option for method gmdistribution.fit. If it is simply adding a 'little' value to diagonal elements of covariance matrix, so as to make covariance ...
8
votes
1answer
1k views

Manifold regularization using laplacian graph in SVM

I'm trying implement Manifold Regularization in Support Vector Machines (SVMs) in Matlab. I'm following the instructions in the paper by Belkin et al.(2006), there's the equation in it: $f^{*} = \...
10
votes
4answers
369 views

Sparsity-inducing regularization for stochastic matrices

It is well-known (e.g. in the field of compressive sensing) that the $L_1$ norm is "sparsity-inducing," in the sense that if we minimize the functional (for fixed matrix $A$ and vector $\vec{b}$) $$f_{...
22
votes
4answers
9k views

L1 regression estimates median whereas L2 regression estimates mean?

So I was asked a question on which central measures L1 (i.e., lasso) and L2 (i.e., ridge regression) estimated. The answer is L1=median and L2=mean. Is there any type of intuitive reasoning to this? ...
1
vote
1answer
116 views

How does the test error pattern depend on the regularizer function?

This question is regarding the role of regularizer in an objective function. Given a loss function $f(x)$, a regularizer function $r(x)$, and $\lambda$ being a trade-off function, our aim is to $\...
21
votes
3answers
6k views

Why do Lars and Glmnet give different solutions for the Lasso problem?

I want to better understand the R packages Lars and Glmnet, which are used to solve the Lasso problem: $$min_{(\beta_0 \beta) \...
11
votes
1answer
310 views

What are $\ell_p$ norms and how are they relevant to regularization?

I have been seeing a lot of papers on sparse representations lately, and most of them use the $\ell_p$ norm and do some minimization. My question is, what is the $\ell_p$ norm, and the $\ell_{p, q}$ ...