Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
1,419
questions
8
votes
1
answer
2k
views
Grid fineness and overfitting when tuning $\lambda$ in LASSO, ridge, elastic net
I wonder about
the optimal grid fineness and
what the relation between grid fineness and overfitting is
in regularization methods such as LASSO, ridge regression or elastic net.
Suppose I want ...
- 67.5k
0
votes
1
answer
467
views
Is adding log-likelihood penalty into optimization problem equivalent to adding prior on model parameters? (why not - in the question)
If we search for a MAP of some $p(\theta \vert D )$ that would look like
$\theta^{MAP} = \arg \max_{\theta} \frac{p(D \vert \theta ) p(\theta)}{Z} = \arg \max_{\theta} p(D \vert \theta ) p(\theta) = \...
- 283
1
vote
1
answer
472
views
Cross validation for determining likelihood penalization in multi-model inference
I looked around Cross Validated and I found pieces of information, but could not answer my particular issue.
I am working with data on detection of an animal species at different locations and ...
- 21
3
votes
0
answers
164
views
Penalized methods comprehensive overview
For the last 10 years from 2004 we have seen a growth in the number of different regularization techniques that have been in use. First it was LASSO, then Adaptive-LASSO, Elastic Net, SCAD, MC+, just ...
78
votes
6
answers
50k
views
Why is multicollinearity not checked in modern statistics/machine learning
In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use ...
- 781
3
votes
1
answer
175
views
What's the current thinking on selecting model complexity in the statistical community?
I was watching a recent presentation by a neural networks researcher who recommended using a model more complex than would be suggested by the data, and regularizing the life out of it. He said this ...
- 2,638
5
votes
1
answer
3k
views
Should the lambda of ridge regression be related to number of data points?
Suppose we have data $ (x_{1}, y_{1})\ldots (x_{N}, y_{N})$. The loss function of ridge regression is
$$
\sum_i^N{(y_i - x^T_i\mathbf{\beta})^2} + \lambda \sum_j^p{\beta^2_j}
$$
Notice that $ \...
- 51
0
votes
1
answer
281
views
Finding all relationships from 500 independent variables to 400 dependent variables (time series)
I am interested in finding statistically significant relationships from a set of 500 independent variables X (actually about 25 variables + their 25 annual moving average x 10 monthly lags) to a set ...
- 163
1
vote
0
answers
90
views
Sequential Least Squares for Tikhonov Regularization [duplicate]
Given a Weighted Linear Least Squares problem where the cost function is given by:
$$ J = { \left( x - H \Theta \right) }^{T} {C}^{-1} { \left( x - H \Theta \right) } $$
There is a Sequential ...
- 1,143
7
votes
1
answer
929
views
Shrinkage of the Sample Covariance matrix
Assume we have N independent and identically distributed random vectors $X_1, X_2, ..., X_N$ where each of them is of size p $\times$ 1. The sample covariance matrix, denoted here by $S$, is computed ...
- 775
91
votes
6
answers
68k
views
Why is the L2 regularization equivalent to Gaussian prior?
I keep reading this and intuitively I can see this but how does one go from L2 regularization to saying that this is a Gaussian Prior analytically? Same goes for saying L1 is equivalent to a Laplacean ...
- 1,249
1
vote
0
answers
91
views
How does one extend radial basis function (RBF) networks formally from regularization but with vector valued outputs?
I was reading the following paper on hyper & radial basis function (HBFs & RBFs) networks and also this one that kind of summarizes the first one and was trying to understand how to extend ...
- 6,866
3
votes
3
answers
6k
views
Is setting lambda equal to zero the same thing as not applying regularization at all?
If I set the regularization parameter to 0, does it essentially mean I'm not applying regularization (I've boxed the regularization bits in red)?
Also, what is this type of regularization called?
- 239
1
vote
1
answer
67
views
Robustness to deviation from normality with regularized VAR model - references
I was listening to a talk where the presenter was talking about using regularized estimation approaches in a VAR(1) model $$X_t = \Gamma X_{t-1} + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0,\...
- 1,880
1
vote
1
answer
72
views
Covariance Estimation for cauchy noise
Apart from Ledoit wolf shrinkage technique which can be better shrinkage covariance estimators for data with cauchy distribution?
How to calculate optimum shrinkage intensity for data containing ...
- 463
5
votes
1
answer
2k
views
The name of 'Fused' Lasso
As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005.
However, I don't get to the meaning of how it is called.
Could anyone give any ...
- 538
7
votes
2
answers
259
views
How to prove the properties of penalized likelihood estimator in Fan and Li (2001) paper
I'm reading through Fan and Li (2001) Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties. On p. 1349 (near the bottom-right corner) they proposed three properties that a ...
- 2,292
1
vote
1
answer
1k
views
Weight shrinking in linear regression by L2 regularization
Quoting Prof. Bengio from his Deep Learning text (http://www.iro.umontreal.ca/~bengioy/dlbook/regularization.html),
$ w = (X^{T}X + \alpha I)^{-1}X^{T}y $
We can see L2 regularization causes ...
2
votes
0
answers
129
views
Shrinkage of Schafer and Strimmer
As we all know that the sample covariance matrix $(S = (s_{ij}))$ is postive definite when the number of observations is smaller than the number of samples, that is n>p.
But, the sample covariance ...
- 775
2
votes
1
answer
3k
views
Early stopping for CNN to improve speed of training
I want to implement early stopping for my convolutional neural network. The main reason is that I want to test my CNN using various parameter settings and some of these may require more iterations ...
- 5,056
2
votes
1
answer
198
views
Can one derive Radial Basis Functions (RBFs) with movable centers from Tikhonov regularization?
It is well know that the "usual" Radial Basis Function can be derived from Regularization that imposes small derivates. More precisely it is well known that the following:
$$ f(x) = \sum^{N}_{n=1} ...
- 6,866
2
votes
0
answers
2k
views
Regularized (LASSO) probit regression
I have a binary variable Y that is a dichotomization of an unknown latent variable, generated by a regression model with normal error. Therefore it makes sense to fit a probit model to Y.
R enables me ...
- 21
21
votes
2
answers
1k
views
confidence intervals' coverage with regularized estimates
Suppose I'm trying to estimate a large number of parameters from some high-dimensional data, using some kind of regularized estimates. The regularizer introduces some bias into the estimates, but it ...
- 11.7k
2
votes
0
answers
3k
views
Interpreting results output from Firth logistic regression in R [duplicate]
I am using Firth logistic regression to analyze data with a rare event. In my model I have 4 continuous variables and 1 dichotomous variable. This is my code:
...
- 21
2
votes
1
answer
731
views
Most Parsimonious Elastic Net Model - choosing $\alpha$ and $\lambda$
How do I calculate which Elastic Net model is the most regularized/parsimonious?
I am recreating GLMnet in another language as an exercise. I want to do a grid search over several values of alpha and ...
- 305
2
votes
0
answers
341
views
Use L1 feature selection when L2 is better fit?
I've run L1 and L2 logistic regression models on my (very large) data and L2 is notably better at ROCAUC. However, depending on the parameters (regularization, class weight) the feature coefficients ...
- 2,061
4
votes
1
answer
3k
views
When the sample covariance matrix becomes singular
Assume a data set $X$ which contains $k$ iid random vectors of size $p$. Denote by $S$ the sample covariance matrix.
Really I have some questions and I need your very appreciated opinions:
1) Ledoit ...
- 775
1
vote
0
answers
34
views
Is there an supervised learning method (a classifier) that can account for unobserved heterogeneity like a mixed logit can?
I'm just starting to teach myself various machine learning techniques. My background is in more "classical" statistics. I've got an analysis that I've done using a mixed logit to predict linkage in ...
- 13.3k
7
votes
1
answer
992
views
Shrinkage of the eigenvalues
Assume we have $n$ samples $X_1,..., X_n$ which are independent and identically distributed with mean = 0 and unknown non-singular covariance matrix $M$. Each sample $X_i$ is a vector of size $p\times ...
- 775
2
votes
1
answer
406
views
What is the prediction equation for penalized logistic regression?
I have used penalized logistic regression (R package logistf) to predict probability of a rare event. 0.12% is the event rate i.e., only 35 occurrence of event in ...
2
votes
0
answers
604
views
Regularization parameter to generate inverse covariance matrix
My data consists of approx. 5 Million binary strings (n) and every string is 2788 characters long. My goal is to find out if position i is correlated with position j. I approximated a covariance ...
5
votes
1
answer
651
views
Stein's estimator vs James-Stein estimator
I read a lot of sources concerning stein's estimator and James-Stein estimator. Unfortunately, a lot of sources do not write the correct formulas of each estimator. And so I am now confused!!
Kindly, ...
- 775
5
votes
1
answer
376
views
Shrinkage estimation of Efron and Morris (1972)
I read this article: Article1 and which was refined by the second article Article2 that was considered as a generalization of the James-Stein estimator.
In article 1 for example, they considered the ...
- 775
6
votes
1
answer
443
views
Why non-negative regression?
I've seen this as regularization technique: impose that the coefficients are non-negative. When is this a good idea? What's the intuition and logic behind it?
- 595
5
votes
3
answers
13k
views
How to use cross-validation with regularization?
I think I understand each of these concepts (cross-validation, regularization) independently, but I'm not quite clear on how they can be put together in practice.
Loosely speaking, in cross-...
- 747
31
votes
5
answers
24k
views
Why is regularisation not applied to bias units in neural networks?
According to this tutorial on deep learning, weight decay (regularization) is not usually applied to the bias terms b why?
What is significance (intuition) behind it?
- 411
36
votes
2
answers
22k
views
What is "reduced-rank regression" all about?
I have been reading The Elements of Statistical Learning and I could not understand what Section 3.7 "Multiple outcome shrinkage and selection" is all about. It talks about RRR (reduced-rank ...
- 9,157
22
votes
1
answer
6k
views
Regularization for ARIMA models
I am aware of LASSO, ridge and elastic-net type of regularization in linear regression models.
Question:
Can this (or a similar) kind of penalized estimation be applied to ARIMA modelling (with a non-...
- 67.5k
5
votes
2
answers
1k
views
Penalizing the Ordinary Least Squares estimation
In a regression analysis, we aim to find the best relationship between two variables (independent variable denoted $y$ and other dependent variable denoted by $x$, and which are related by: $y = f_\...
- 775
30
votes
5
answers
15k
views
Do we still need to do feature selection while using Regularization algorithms?
I have one question with respect to need to use feature selection methods (Random forests feature importance value or Univariate feature selection methods etc) before running a statistical learning ...
- 1,115
9
votes
1
answer
2k
views
Regularization and projection onto the $l_*$ ball
I'm trying to understand how regularization works in term of projections onto a $l_*$ ball, and Euclidean projection onto the simplex.
I'm not sure I understand what we mean when we project the ...
- 2,862
1
vote
1
answer
1k
views
Maximum and minimum penalty in lasso regression family
I am trying to adjust the penalty $\lambda$ in group lasso regression, but I have no idea about it. Just to clarify, group lasso regression is a kind of multiple linear regression which use penalties ...
- 502
0
votes
0
answers
81
views
Evaluating integral to obtain marginal PDF related to Tikhonov Regularization
I am attempting to derive the marginal PDF for an application of the Gibbs Sampler. My joint PDF contains:
$P(b,x) = \frac{1}{\sigma^{n}}\exp \left( -\frac{1}{2\sigma^2}\left\lVert b-Ax\right\rVert^2-...
- 111
7
votes
1
answer
2k
views
Regularization in Neural networks
One way to regularize a neural network is "early stopping" , meaning that I don't let the weights get to their optimal values (based on the cost function calculated on the training data) but stop the ...
- 398
2
votes
1
answer
422
views
Least-squares training error
In classification problems, the training error typically decreases as further training examples are acquired. However, in my current least-squares problem, the training error actually increases as ...
- 7,039
2
votes
0
answers
308
views
Neural Network Learning Curves with Low Test Set Error
TLDR: You can see my neural network learning curves here: https://i.stack.imgur.com/hBbS7.jpg. Which regularization term would you pick given that the test error actually drops below the training ...
1
vote
1
answer
837
views
For a quadratic form to minimize with a L2 regularization term, is the gradient of the solution collinear to the solution?
Say you minimize a quadratic form f with a L2 regularization term (g = f + L2_term). The solution of minimizing g is x*. Is the gradient of f applied to x* collinear to x* as the figure below suggests?...
- 418
7
votes
1
answer
1k
views
How to choose the regularization parameter in ZCA whitening?
ZCA whitening can use regularization, as in
$$
\tilde{X} = L\sqrt{(D + \epsilon)^{-1}}L^{-1}X,
$$
where $LDL^\top$ is an eigendecomposition of the sample covariance matrix. What's a good choice for ...
- 1,327
9
votes
1
answer
2k
views
Range of lambda in elastic net regression
$\def\l{|\!|}$
Given the elastic net regression
$$\min_b \frac{1}{2}\l y - Xb \l^2 + \alpha\lambda \l b\l_2^2 + (1 - \alpha) \lambda \l b\l_1$$
how can an appropriate range of $\lambda$ be chosen ...
- 3,682
4
votes
0
answers
478
views
How to use glmnet with panel data?
I would like to use regularization from the package glmnet on my panel data. I have the following data: country - year - gdp - population - water usage, so for each country, I have a year in which the ...
- 1,185