Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

3
votes
1answer
60 views

what does regularization mean in xgboost (tree)

In xgboost (xgbtree), gamma is the tunning parameter to control the regularization. I understand what regularization means in <...
0
votes
0answers
10 views

Understanding the MC dropout (Monte Carlo estimate)

I've recently came across the term "MC dropout" in one of the papers I was reading. In Neural Networks, when using a standard dropout layer on top of one of the model's layers, during training the ...
3
votes
0answers
35 views

Why does cv.glmnet not use the same lambda sequence across different folds to find the hypertuning parameter lambda?

I assumed that cv.glmnet works as follows: Generate multiple glmnet fits for the entire data, presumably for automated lambda sequence using coordinate descent Use the lambdas gotten in step 1, and ...
0
votes
0answers
16 views

How dose dropout affect Weights and Bias?

I applied dropout in my network , and it worked , but i can't interpret dropout effects on weight and bias, to be more specific , i can't interpret why appling droput and not applying dropout have a ...
6
votes
1answer
122 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
0
votes
0answers
9 views

How to find AUC for Precision-recall curve using glmnet in R? [closed]

I am running penalized regression in R on an imbalanced dataset using glmnet. Since glmnet masks all other AUC functions in other packages, Can anybody help on how to find Precision-Recall AUC using ...
2
votes
0answers
22 views

$L^2$ Regularization and Hessian Matrix [duplicate]

In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
5
votes
1answer
62 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
2
votes
0answers
27 views

In regularised regression, why not penalise higher degree coefficients more?

for the case of fitting a polynomial to data ($y = a_{1}x+a_{2}x^{2}+...+a_{n}x^{n}$) with regularisation of the coefficients, I'm wondering why are all of the coefficients usually penalised by the ...
1
vote
1answer
26 views

Why increasing lambda parameter in L2-regularization makes the co-efficient values converge to zero [duplicate]

Why increasing lambda parameter in L2-regularization makes the co-efficient values converge to zero? I have just tried to do the math, but it's a little bit rusted. Lets say that we have a simple ...
0
votes
0answers
15 views

Implicit regularization in Linear models

Regarding Linear Neural Networks models with unique finite root loss function, without an explicit regularization, I am struggling to prove that in the case of overparmeterized models (i.e. $N<d$), ...
0
votes
0answers
16 views

Selection criteria for penalty parameters in the ridge multinomial logit model

I appeal to you for the following doubt. I am adjusting a ridge multinomial logit model but I have problems in the criterion when choosing the lambda parameter that gives better results, besides ...
0
votes
0answers
31 views

Regularization of linear regression problem [duplicate]

Consider a vector $a \in R^n$. I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where $f(x) = ||x-a||_{2}^2 + \lambda ||x|...
0
votes
0answers
16 views

SVM training yields too many (or no) support vectors

So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine. However, ...
0
votes
0answers
26 views

How is the generalization of LASSO called?

I know that ridge regression is a special case of Tykhonv regularization. In fact with Tykhonov one tries to minimize: $|| Ax - b ||^2 +|| \Gamma x ||^2$ If $\Gamma$ is the identity matrix scaled by ...
0
votes
1answer
19 views

Data augmentation methods for Raman Spectra

I'm building a CNN model based on Raman spectroscopy data and I wanted to experiment with data augmentation. What would be some reasonable techniques to try? I have found this paper which suggests ...
0
votes
0answers
27 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
81 views

Can one usefully specify a multilevel-model with a partially-nested, partially non-nested structure?

Background Gelman and Hill's Data Analysis Using Regression and Multilevel/Hierarchical Models includes an example in section 13.5 of how to model non-nested data. The second example in this section ...
2
votes
1answer
19 views

What value of alpha should I choose regularization

What value of alpha should I choose in glmnet? Should I use one which minimizes the cross-validation error, one which is one standard deviation above or below the one which gives the best error (like ...
0
votes
0answers
21 views

Adding regularization to an objective function when not using gradient descent

Using a simple example if I have a model: $$y = \beta_1 X_1 + \beta_2X_2 + {\rm error}$$ with cost function $${\rm Cost}= RSS + \alpha (\beta_1 + \beta_2)(\beta_1 + \beta_2)$$ If we were to use ...
0
votes
0answers
17 views

caret chooses non-optimal RMSE?

I run a linear regression via caret / glmnet method with "RMSE" as metric. In the final model, caret tells me which values of the tuning parameters alpha and lambda were selected to minimize RMSE. If ...
1
vote
0answers
128 views

Whether to use dropout vs. batch normalization vs L1/L2 loss for regularization

I am familiar with how dropout, batch normalization, and L1/L2 loss all work. However, I do not have an intuitive sense on when to use which. There are lots of discussion between dropout and batch ...
0
votes
1answer
29 views

How to regularize estimated probabilities in linear regression?

I estimate a linear probability model of the form: $ E[y\mid X] = X\beta $ where $y$ is a binary variable (hence, $E[y\mid X]= Pr(y=1\mid X)$) and $X$ a matrix coding a categorical variable by ...
4
votes
0answers
57 views

Do we have the actual theoretical study of L1/L2 regularization for Logistic regression?

It is very well known that L1 and L2 regularization can help in reducing the generalization error, and their effectiveness has been empirically demonstrated across a large set of machine learning ...
1
vote
1answer
185 views

Difference between kernel, bias, and activity regulizers in Keras

I've read this post, but I wanted more clarification for a broader question. In Keras, there are now three types of regularizers for a layer: kernel_regularizer, <...
1
vote
0answers
19 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
0
votes
1answer
74 views

Total variation regularization in deep learning

For current deep learning models, we can find basically two kinds of regularization on: Activation Weights The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization ...
0
votes
0answers
19 views

Ridge analytically vs glmnet [duplicate]

With an outcome variable and two correlated regressors... ...
0
votes
0answers
24 views

How to avoid L1 regularization causing informative features to get a weight of exactly 0.0.?

L1 regularization may cause the following kinds of features to be given weights of exactly 0: Weakly informative features. Strongly informative features on different scales. Informative features ...
1
vote
0answers
33 views

Bayesian Regression with LASSO

I am trying to build a Bayesian regression model with LASSO regularization. My understanding is that I can do this by setting a Laplace prior on the coefficients. I also need a prior for the variance ...
2
votes
0answers
153 views

Difference Between Two Tikhonov Regularization Schemes

For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions: $x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below $x = (A^TA + \...
1
vote
1answer
39 views

Is there any library for least absolute deviation (LAD) regression with regularization terms?

We know that LASSO and ridge and ElasticNet all apply regularization terms on the coefficients of least squares regression. However, I have not yet found any R / python libraries that compute ...
0
votes
0answers
32 views

Regularization in Bayesian updating

I'm building an online algorithm (I have streaming data) that does Bayesian Linear Regression. Each time data comes in, I use standard Bayesian updating formulae to calculate the posterior, which ...
0
votes
0answers
23 views

which parameter you choose on lasso CV, tuning parameter λ or βi constraint s?

I try to use lasso for prediction and I have $X_{tr} \subset X$ the train set and $Y_{tr}$ the train target. and I have $X_{ts} \subset X$ and $ Y_{ts}$ the test set for CV. I used CV and got $λ_i$ ...
2
votes
0answers
30 views

The limitations of Elastic-net regularization [duplicate]

I know that Elastic-net regularization is the combination of L1 and L2 regularization. My question is what are the limitations of Elastic-net regularization? My question is not related to Elastic-...
5
votes
0answers
29 views

Equations of Elastic net regularization [duplicate]

I know that the Elastic net takes care of the limitations of Lasso by adding an L 2 penalty term. In the attached picture has been mentioned that the two formulas are equivalent. I tried to show this, ...
1
vote
0answers
81 views

How does Ridge Regression penalize for complexity if the coefficients are never allowed to go to zero?

In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas: Both of these methods attempt to improve generalization ...
1
vote
1answer
534 views

L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
2
votes
1answer
256 views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
2
votes
1answer
37 views

Is constrained (nonnegative) least squares a form a regularisation?

Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
3
votes
0answers
75 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
1
vote
1answer
44 views

How we can avoid making L2 regularization causing the model to learn a moderate weight for some non-informative features.?

Referencing to an example explained in free google machine learning course Imagine a linear model with 100 input features: 10 are highly informative. 90 are non-informative. Assume that all ...
0
votes
0answers
10 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
7
votes
2answers
66 views

I regularized my linear regression, now what?

I have estimated the regression parameters of a linear regression models using LASSO, sent some variables to zero using cross validation, and now I got a final model. It is known that regularizing ...
0
votes
1answer
128 views

Does dropout regularization prevent overfitting due to too many iterations?

For image classification problem, let's say, and given a neural network to train on, if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
0
votes
2answers
60 views

L1 L2 regularization [duplicate]

The tutorial says the intersection point for L1 and L2 regularization gives the minimum loss - But why the intersection gives the minimum loss? I cannot interpret the graph clearly.
2
votes
2answers
196 views

How to prevent overfitting in Gaussian Process

I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data. I tried different kernels and find the following kernel (2 RBF + a white noise)...
1
vote
0answers
31 views

RKHS norm and Fourier transform link

In the notes here, it is stated that norms of some reproducing kernel Hilbert spaces can be written in terms of Fourier transforms, and this is often used to argue that a higher RKHS norm implies a ...
0
votes
0answers
63 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
2
votes
1answer
214 views

Ridge/Lasso regression negative Lambda

I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...