Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

0
votes
1answer
25 views

Difference between kernel, bias, and activity regulizers in Keras

I've read this post, but I wanted more clarification for a broader question. In Keras, there are now three types of regularizers for a layer: kernel_regularizer, <...
1
vote
0answers
18 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
0
votes
1answer
30 views

Total variation regularization in deep learning

For current deep learning models, we can find basically two kinds of regularization on: Activation Weights The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization ...
0
votes
0answers
15 views

Ridge analytically vs glmnet [duplicate]

With an outcome variable and two correlated regressors... ...
0
votes
0answers
20 views

How to avoid L1 regularization causing informative features to get a weight of exactly 0.0.?

L1 regularization may cause the following kinds of features to be given weights of exactly 0: Weakly informative features. Strongly informative features on different scales. Informative features ...
1
vote
0answers
22 views

Bayesian Regression with LASSO

I am trying to build a Bayesian regression model with LASSO regularization. My understanding is that I can do this by setting a Laplace prior on the coefficients. I also need a prior for the variance ...
2
votes
0answers
119 views

Difference Between Two Tikhonov Regularization Schemes

For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions: $x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below $x = (A^TA + \...
0
votes
0answers
16 views

Asymptotics for Lasso-Type Estimators

In the paper Asymptotics for Lasso-Type Estimators, https://projecteuclid.org/euclid.aos/1015957397 the authors study the asymptotic properties of the LASSO estimators. I am confused, how can we ...
1
vote
1answer
25 views

Is there any library for least absolute deviation (LAD) regression with regularization terms?

We know that LASSO and ridge and ElasticNet all apply regularization terms on the coefficients of least squares regression. However, I have not yet found any R / python libraries that compute ...
0
votes
0answers
29 views

Regularization in Bayesian updating

I'm building an online algorithm (I have streaming data) that does Bayesian Linear Regression. Each time data comes in, I use standard Bayesian updating formulae to calculate the posterior, which ...
0
votes
0answers
19 views

which parameter you choose on lasso CV, tuning parameter λ or βi constraint s?

I try to use lasso for prediction and I have $X_{tr} \subset X$ the train set and $Y_{tr}$ the train target. and I have $X_{ts} \subset X$ and $ Y_{ts}$ the test set for CV. I used CV and got $λ_i$ ...
2
votes
0answers
27 views

The limitations of Elastic-net regularization [duplicate]

I know that Elastic-net regularization is the combination of L1 and L2 regularization. My question is what are the limitations of Elastic-net regularization? My question is not related to Elastic-...
5
votes
0answers
29 views

Equations of Elastic net regularization [duplicate]

I know that the Elastic net takes care of the limitations of Lasso by adding an L 2 penalty term. In the attached picture has been mentioned that the two formulas are equivalent. I tried to show this, ...
1
vote
0answers
38 views

How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?

In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas: Both of these methods attempt to improve generalization ...
1
vote
1answer
337 views

L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
2
votes
1answer
71 views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
2
votes
1answer
29 views

Is constrained (nonnegative) least squares a form a regularisation?

Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
3
votes
0answers
57 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
1
vote
1answer
37 views

How we can avoid making L2 regularization causing the model to learn a moderate weight for some non-informative features.?

Referencing to an example explained in free google machine learning course Imagine a linear model with 100 input features: 10 are highly informative. 90 are non-informative. Assume that all ...
0
votes
0answers
8 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
6
votes
2answers
60 views

I regularized my linear regression, now what?

I have estimated the regression parameters of a linear regression models using LASSO, sent some variables to zero using cross validation, and now I got a final model. It is known that regularizing ...
0
votes
1answer
39 views

Does dropout regularization prevent overfitting due to too many iterations?

For image classification problem, let's say, and given a neural network to train on, if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
0
votes
2answers
49 views

L1 L2 regularization [duplicate]

The tutorial says the intersection point for L1 and L2 regularization gives the minimum loss - But why the intersection gives the minimum loss? I cannot interpret the graph clearly.
0
votes
0answers
24 views

Cyclic coordinate descent in R clogitL1 package

I refer to the paper by Reid & Tibshirani, which is freely available via this link and details the cyclic coordinate descent algorithm behind the R clogitL1 package. By cycling through $j=1,...,p$...
0
votes
0answers
18 views

Regularization parameter in LARS

I'm reading Elements of Statistical Learning book and trying to find out how LASSO modification of LARS algorithm works. I don't quite understand where the regularization parameter $\lambda$ appears ...
1
vote
2answers
84 views

How to prevent overfitting in Gaussian Process

I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data. I tried different kernels and find the following kernel (2 RBF + a white noise)...
1
vote
0answers
17 views

RKHS norm and Fourier transform link

In the notes here, it is stated that norms of some reproducing kernel Hilbert spaces can be written in terms of Fourier transforms, and this is often used to argue that a higher RKHS norm implies a ...
0
votes
0answers
15 views

“exotic” weight regularization?

In neural nets, there is often a weight regularization term in the loss function, which ensures that unnecessarily high weights don't occur. For example, if $C(\theta,x)$ is the baseline loss function,...
0
votes
0answers
62 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
2
votes
1answer
126 views

Ridge/Lasso regression negative Lambda

I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
38 views

Lasso vs. Linear Regression with features selected by lasso: what to expect

I cross validate a lasso regression with multiple values of lambda (the multiplier for the penalty) e.g. from 0.00001 to 100 I get the best solution is with a certain lambda, e.g. 0.7 Given some of ...
0
votes
0answers
31 views

How to penalize change of states in Hidden Markov model?

I'm trying to fit a HMM on a sequence of observations and I would like to introduce some constraints that would penalize an excessive number of changes of state in the complete sequence (where "change"...
0
votes
0answers
18 views

What causes a high testing deviance vs. training deviance in a gradient boosting classifier?

My main goal is to classify multi-class data using supervised learning. Currently, I am looking into GradientBoostingClassifier as the estimator. I want to make sure I am selecting the model ...
0
votes
1answer
32 views

Adding more samples to ordinary regression is equall to ridge regression [duplicate]

I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
2
votes
0answers
25 views

Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified?

Could it make sense (and if so, under what circumstances) to define a penalized estimator based on one loss function but then select its tuning parameter (say, via cross validation) based on another ...
1
vote
0answers
11 views

How is the location of this point derived?

Given a vector $\mathbf{w_{\theta}}$ which is normal to the orange line: And two point $\mathbf{x}$ how is $\mathbf{x_p}$ derived? I understand that the vector $\mathbf{w_{\theta}}$ is the vector ...
4
votes
0answers
81 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
1
vote
0answers
36 views

Intuitive definition of manifold regularization for neural networks [closed]

I am studying the deep neural networks and I have been assigned a project on the manifold regularized neural networks, in particular the definition is as follows: "enabling semi-supervised learning ...
0
votes
1answer
154 views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
5
votes
1answer
174 views

MAP estimation as regularisation of MLE

Going through the Wikipedia article on Maximum a posteriori estimation, it got confusing after reading this: It is closely related to the method of maximum likelihood (ML) estimation, but employs ...
1
vote
0answers
17 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
17 views

What can be implied from loss function that its regularizer needs large coefficient

I run loss function with l1-norm as regularizer for source separation. $min\sum_{i=1}^{n} V(f(x_{i}), y_{i}) + \lambda R(f)$ I varied the coefficient ($\lambda$) from 0 to 1e14. The results ($\frac{\...
0
votes
0answers
13 views

What is difference in feature selection by using l0 and l1 regularization?

Both l0 and l1 can be used for feature selection, so what is the difference between them?
2
votes
2answers
67 views

Variables reduction required for Random Forest, Boosting, L1, L2 regularization

I have close to 10,000 variables. I know how random forest/XGB picks number of variables randomly for building the tree. Also regularization takes care of significance of variable by its coefficient. ...
11
votes
4answers
500 views

What causes lasso to be unstable for feature selection?

In compressed sensing, there is a theorem guarantee that $$\text{argmin} \Vert c \Vert_1\\ \text{subject to } y = Xc $$ has a unique sparse solution $c$ (See appendix for more details). Is there a ...
0
votes
1answer
48 views

Does regularization leads to stucking in local minima?

I frequently hear some very conflicting claims regarding deep learning algorithms. Currently, I am a bit confused on the role of regularization. I have listed my queries below regarding regularization ...
0
votes
1answer
25 views

Regression regularization penalty center at w0 instead of 0 [duplicate]

How do I regress with regularization penalty term lambda * (w - w0)^2 instead of lambda * w^2? Is there any package to do it? I ...
1
vote
1answer
73 views

$\lambda \Vert k \Vert_0$ or $\Vert k \Vert_0 \leqslant n$

Say $Y \in \Bbb R^n$ is a response, $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ are predictors. In a linear regression problem, we want to add an $l_0$ regularization for feature selection....
6
votes
0answers
121 views

Sparse linear regression 0-norm and 1-norm

We have a response $Y \in \Bbb R^n$ and predictors $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ The problem we want to solve is $$\text{argmin}_{k \in \Bbb R^{m}} (\Vert Y - Xk \Vert_2^2 +...
0
votes
0answers
16 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...