Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
1,420
questions
0
votes
0
answers
51
views
How to give a represantation to veriables from each group using LASSO
I'm trying to apply LASSO regression on my data set in order to choose the best variables. However, my variables (44 to be accurate) come from 7 different groups, is there any option to give a "...
- 11
4
votes
0
answers
366
views
Do we have the actual theoretical study of L1/L2 regularization for Logistic regression?
It is very well known that L1 and L2 regularization can help in reducing the generalization error, and their effectiveness has been empirically demonstrated across a large set of machine learning ...
user42004
35
votes
3
answers
30k
views
What is the difference between kernel, bias, and activity regulizers, and when to use which?
I've read this post, but I wanted more clarification for a broader question.
In Keras, there are now three types of regularizers for a layer: kernel_regularizer, <...
- 1,872
1
vote
0
answers
36
views
To remove neural-network units or to increase drop-out?
When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
- 1,120
1
vote
1
answer
3k
views
Total variation regularization in deep learning
For current deep learning models, we can find basically two kinds of regularization on:
Activation
Weights
The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization ...
- 420
2
votes
1
answer
199
views
Soft thresholding (Donoho and Johnstone)
Donoho and Johnstone (1994) poses the following equality:
$$
E((\eta_t(X) - \mu)^2) = 1 - 2\Pr(|X|\lt t) + E(\min(X^2,t^2))
$$
where $\eta_t(X) = \operatorname{sign}(x)\max(|X|-t,0)$ and $X \sim N(\...
- 133
0
votes
0
answers
50
views
Ridge analytically vs glmnet [duplicate]
With an outcome variable and two correlated regressors...
...
- 1,086
1
vote
1
answer
227
views
Encoding variable number of categorical features
I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories.
There are over ...
- 240
0
votes
0
answers
122
views
How to avoid L1 regularization causing informative features to get a weight of exactly 0.0.?
L1 regularization may cause the following kinds of features to be given weights of exactly 0:
Weakly informative features.
Strongly informative features on different scales.
Informative features ...
- 673
1
vote
0
answers
75
views
Bayesian Regression with LASSO
I am trying to build a Bayesian regression model with LASSO regularization. My understanding is that I can do this by setting a Laplace prior on the coefficients. I also need a prior for the variance ...
- 681
0
votes
0
answers
151
views
Shrinkage methods - are they any good for statistical inference or should they be used for prediction goals only?
I am working on my master thesis with a goal to find regressors which influence companies' decisions on how to pay for a target in acquisitions (cash, stock or a mix of both).
I have 13 regressors to ...
- 125
1
vote
0
answers
144
views
Removing the intercept term for penalized logistic regression
I am working on lasso logistic regression and am trying to remove the intercept term from the penalty function. I have tried to use the mean centering theory but for logistic regression it can not be ...
2
votes
0
answers
432
views
Difference Between Two Tikhonov Regularization Schemes
For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions:
$x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below
$x = (A^TA + \...
- 4,486
3
votes
1
answer
329
views
Large value of $X\beta$ in logistic regression?
In logistic regression, the probability is obtained from
$$
Pr = \frac{\exp(X\beta)}{1 + \exp(X\beta)} ~~~~ (1)
$$
From the plot below, it is obvious that if $X\beta$ > 10, the probability approaches ...
- 449
5
votes
3
answers
4k
views
Is there any library for least absolute deviation (LAD) regression with regularization terms?
We know that LASSO and ridge and ElasticNet all apply regularization terms on the coefficients of least squares regression. However, I have not yet found any R / python libraries that compute ...
- 51
0
votes
0
answers
54
views
Regularization in Bayesian updating
I'm building an online algorithm (I have streaming data) that does Bayesian Linear Regression.
Each time data comes in, I use standard Bayesian updating formulae to calculate the posterior, which ...
- 688
2
votes
0
answers
81
views
The limitations of Elastic-net regularization [duplicate]
I know that Elastic-net regularization is the combination of L1 and L2 regularization.
My question is what are the limitations of Elastic-net regularization?
My question is not related to Elastic-...
- 2,089
5
votes
0
answers
74
views
Equations of Elastic net regularization [duplicate]
I know that the Elastic net takes care of the limitations of Lasso by adding an L 2 penalty term. In the attached picture has been mentioned that the two formulas are equivalent. I tried to show this, ...
- 2,089
0
votes
1
answer
80
views
Confidence limits for constrained penalized log likelihood model
I am estimating parameter $\beta$ as:
\begin{align}
\hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
- 113
5
votes
1
answer
2k
views
How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?
In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas:
Both of these methods attempt to improve generalization ...
- 11.9k
5
votes
1
answer
4k
views
L1 and L2 penalty vs L1 and L2 norms
I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models.
From what I understand:
L1: Laplace Prior L2: Gaussian Prior
are two of the ...
- 211
12
votes
1
answer
10k
views
Why L1 regularization can "zero out the weights" and therefore leads to sparse models? [duplicate]
I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1].
To better understand it I'm reading Google's ...
- 2,395
4
votes
1
answer
836
views
Is constrained (nonnegative) least squares a form a regularisation?
Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
- 9,157
4
votes
0
answers
970
views
Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions? [duplicate]
While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says:
Without regularization, the asymptotic nature of logistic regression i.e (it ...
- 673
2
votes
1
answer
464
views
How we can avoid making L2 regularization causing the model to learn a moderate weight for some non-informative features.?
Referencing to an example explained in free google machine learning course
Imagine a linear model with 100 input features:
10 are highly informative.
90 are non-informative.
Assume that all ...
- 673
7
votes
2
answers
103
views
I regularized my linear regression, now what?
I have estimated the regression parameters of a linear regression models using LASSO, sent some variables to zero using cross validation, and now I got a final model. It is known that regularizing ...
- 73
1
vote
1
answer
7k
views
Does dropout regularization prevent overfitting due to too many iterations?
For image classification problem, let's say, and given a neural network to train on,
if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
- 349
2
votes
2
answers
756
views
L1 L2 regularization [duplicate]
The tutorial says the intersection point for L1 and L2 regularization gives the minimum loss - But why the intersection gives the minimum loss? I cannot interpret the graph clearly.
- 1,087
15
votes
2
answers
10k
views
How to prevent overfitting in Gaussian Process
I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data.
I tried different kernels and find the following kernel (2 RBF + a white noise)...
- 311
0
votes
0
answers
125
views
Adding additional constrains to OpenAi Gym
I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym.
Copied some code from GitHub which isn't deep yet:
<...
- 681
9
votes
1
answer
818
views
RKHS norm and Fourier transform link
In the notes here, it is stated that norms of some reproducing kernel Hilbert spaces can be written in terms of Fourier transforms, and this is often used to argue that a higher RKHS norm implies a ...
- 1,481
4
votes
2
answers
4k
views
Ridge/Lasso regression negative Lambda
I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
- 147
1
vote
0
answers
360
views
Lasso vs. Linear Regression with features selected by lasso: what to expect
I cross validate a lasso regression with multiple values of
lambda (the multiplier for the penalty) e.g. from 0.00001 to 100
I get the best solution is with a certain lambda, e.g. 0.7
Given some of ...
- 85
0
votes
0
answers
264
views
How to penalize change of states in Hidden Markov model?
I'm trying to fit a HMM on a sequence of observations and I would like to introduce some constraints that would penalize an excessive number of changes of state in the complete sequence (where "change"...
- 393
0
votes
1
answer
282
views
Adding more samples to ordinary regression is equall to ridge regression [duplicate]
I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
- 11
3
votes
1
answer
428
views
Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified?
Could it make sense (and if so, under what circumstances) to define a penalized estimator based on one loss function but then select its tuning parameter (say, via cross validation) based on another ...
- 67.6k
3
votes
0
answers
1k
views
Calibration of penalized (LASSO or ELasticNet) logistic regression models
I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome.
I would like my prediction model to be calibrated (mean ...
3
votes
0
answers
410
views
Elastic Net and collinearity
I am performing elastic net for variable selection on a dataset of 95 records and 41 variables. The response is a continuous numerical.
I choose the alpha and lambda parameters through 10 fold cross ...
- 171
2
votes
0
answers
19
views
How is the location of this point derived?
Given a vector $\mathbf{w_{\theta}}$ which is normal to the orange line:
And two point $\mathbf{x}$ how is $\mathbf{x_p}$ derived? I understand that the vector $\mathbf{w_{\theta}}$ is the vector ...
- 1,261
5
votes
0
answers
349
views
Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?
Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
- 11.9k
0
votes
1
answer
1k
views
Elastic net visualization [closed]
Is there a way to display in a graph the elastic net (or penalized regression in general) results?
Specifically, how can I render the coefficients of the variables?
- 171
1
vote
1
answer
4k
views
stochastic gradient descent of ridge regression when regularization parameter is very big
As we know, the gradient of ridge regression is:
$$
g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta
$$
where $X_i$ is the $i$th training sample.
The update of $\theta$ is ...
- 11
9
votes
1
answer
5k
views
MAP estimation as regularisation of MLE
Going through the Wikipedia article on Maximum a posteriori estimation, it got confusing after reading this:
It is closely related to the method of maximum likelihood (ML) estimation, but employs ...
- 1,039
2
votes
0
answers
268
views
Training an ANN further once it reaches 100 % accuracy on training set
I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data?
I'm facing a binary classification problem and read this article ...
- 458
1
vote
0
answers
21
views
What can be implied from loss function that its regularizer needs large coefficient
I run loss function with l1-norm as regularizer for source separation.
$min\sum_{i=1}^{n} V(f(x_{i}), y_{i}) + \lambda R(f)$
I varied the coefficient ($\lambda$) from 0 to 1e14. The results ($\frac{\...
- 171
3
votes
2
answers
2k
views
Variables reduction required for Random Forest, Boosting, L1, L2 regularization
I have close to 10,000 variables. I know how random forest/XGB picks number of variables randomly for building the tree. Also regularization takes care of significance of variable by its coefficient.
...
- 65
15
votes
4
answers
9k
views
What causes lasso to be unstable for feature selection?
In compressed sensing, there is a theorem guarantee that
$$\text{argmin} \Vert c \Vert_1\\
\text{subject to } y = Xc
$$
has a unique sparse solution $c$ (See appendix for more details).
Is there a ...
- 1,832
3
votes
1
answer
2k
views
Does regularization leads to stucking in local minima?
I frequently hear some very conflicting claims regarding deep learning algorithms. Currently, I am a bit confused on the role of regularization. I have listed my queries below regarding regularization ...
- 601
0
votes
1
answer
80
views
Regression regularization penalty center at w0 instead of 0 [duplicate]
How do I regress with regularization penalty term lambda * (w - w0)^2 instead of lambda * w^2? Is there any package to do it?
I ...
- 147
2
votes
1
answer
98
views
$\lambda \Vert k \Vert_0$ or $\Vert k \Vert_0 \leqslant n$
Say $Y \in \Bbb R^n$ is a response, $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ are predictors. In a linear regression problem, we want to add an $l_0$ regularization for feature selection....
- 1,832