Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

2
votes
1answer
33 views

Why is lasso more robust to outliers compared to ridge?

In my attempt to reason about it intuitively I am concluding that ridge might be more robust to outliers. Following is my intuitive/lose reasoning : If there is an outlier then to match my ...
0
votes
1answer
18 views

How can I store information in a custom regularizer? [closed]

I'm trying to create a custom keras regularizer that uses the distance of the layer's weights from it's original weights, but what I used doesn't seem to work. I get a zero difference at all times. ...
4
votes
1answer
33 views

Should MLE estimation always be using penalizers?

I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some ...
0
votes
0answers
17 views

Estimation of Stationary Vector Autoregressive Processes with R or Python [closed]

What well-documented R or Python package would you recommend that deals with penalized estimation of stationary vector autoregressive processes? I am particularly interested in the lasso but open-...
1
vote
0answers
7 views

Does the expected cross validated error/NLL from regularisation have one minimum?

When trying to choose the value of the regularisation parameter(s) in lasso, ridge regression, or elasticnet, one generally computes the cross validated error or negative log-likelihood as explained ...
2
votes
1answer
34 views

How does L2 penalize large weights

The L2 regularization term is useful because it penalizes large weights over smaller weights which is good to prevent overfitting. I'm having a hard time understanding how exactly it does this. This ...
0
votes
0answers
17 views

Link between norm of weights/coefficients and smoothness

We often avoid overfitting by penalizing the norm of the weights/coefficients (in a classic Ridge or Lasso regression). I understand that we want smooth functions as they will be more likely to ...
0
votes
0answers
25 views

How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
0
votes
0answers
21 views

Lasso Logistic Regression in the presence of Class Imbalance

Since class imbalance only affects the estimate of the intercept in vanilla logistic regression, the orientation of the optimal separating hyperplane remains unchanged. However in $L_1$-regularized ...
3
votes
1answer
224 views

Does regularization penalize models that are simpler than needed?

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?
1
vote
1answer
18 views

Consisten Regularizer for Neural Network

In the book 'Pattern Recognition and Machine Learning' by Bishop (p.257 ff.) he considers a weight decay regularizer of the error function $$\hat E(w)=E(w)+\frac{\lambda}{2}w^tw$$ where $w$ is a ...
0
votes
0answers
11 views

Package to perform 2nd degree polynomial regression with L1 penalty for use of the 2nd degree

I'm trying to fit either a straight line or 2nd degree polynomial through many sets of points (2-dimensional data). I would much prefer a straight line over a polynomial, so am trying to penalize the ...
0
votes
0answers
12 views

Finding significant explanatory values with unregularized logistic regression?

So i'm working currently with a colleague on a classification problem where both of us have different approaches. So i try to find the most important features through standard L1 regularization and he ...
1
vote
0answers
27 views

Is there a theoretical basis for using partial least squares with categorical responses

I am using what is called PLS-DA in JMP to find a model for predicting a categorical (Positive/Negative) response. The documentation says that the responses are simply coded as 0/1, thereby ...
2
votes
1answer
120 views

Ridge Regression with Gradient Descent Converges to OLS estimates

I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression. This is ...
0
votes
0answers
19 views

Using Quadratic Programming to solve Lasso and Ridge regression models?

I'm trying to build linear, ridge and lasso regression models for at set of data (40 obs., 4 features, 1 response). I'm building the models using the sklearn package for Python and I can easily find ...
0
votes
1answer
14 views

Interpreting ridge coefficients as a function of regularization

Data consists of 40 observations with 4 dimensions and a response-variable. When doing a ridge regression on my data and plotting the coefficients and coefficient errors (MSE of the ridge ...
0
votes
0answers
27 views

How to calculate One Standard Error rule tuning parameter for prediction error during k-fold CV

I'm trying to wrap my head around exactly how this rule goes into place, so I can use it by hand in other model selection setups. So here's some R code to get it started: ...
0
votes
0answers
7 views

choice of the lambda parameter in the logit multinomial ridge model

I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
0
votes
0answers
44 views

Random partition techniques for lasso and elastic net

I think this is the correct place for this question. I have implemented lasso, elastic net and a different estimator on a real data set. I used 10-fold cross-validation (CV) one time without ...
0
votes
0answers
23 views

Why does Group Lasso use L2 norm for individual group penalties?

In group lasso $$\min_{\beta}\left\{\frac{1}{2} \left\lVert{y}-\sum_{l=1}^mX^{(l)}{\beta^{(l)}}\right\rVert_2^2 +\lambda\sum_{l=1}^m\sqrt{p_l}\left\lVert{\beta^{(l)}}\right\rVert_q\right\}$$ the ...
2
votes
0answers
47 views

Probability that feature selection in elastic net regularisation is meaningful - evaluating the statistical significance of chosen features

I have a research question - can I use baseline clinical features to predict my binary clinical outcome in individual patients? I am interested if the performance of my model is greater than chance. I ...
1
vote
0answers
22 views

Which metric should be used for learning rate reduction on plateau?

I am testing three different neural network architectures on a dataset to see which architecture performs the best. My methodology for every architecture is to Split the data into train/test Take ...
1
vote
1answer
61 views

Convex optimization: Is gradient descent faster if a regularizer is added?

I am not sure if this is a true statement or not but there appears to be an intuition around this among experts in the field that I do not quite understand. The idea is: Given a convex optimization ...
3
votes
1answer
138 views

what does regularization mean in xgboost (tree)

In xgboost (xgbtree), gamma is the tunning parameter to control the regularization. I understand what regularization means in <...
3
votes
0answers
69 views

Why does cv.glmnet not use the same lambda sequence across different folds to find the hypertuning parameter lambda?

I assumed that cv.glmnet works as follows: Generate multiple glmnet fits for the entire data, presumably for automated lambda sequence using coordinate descent Use the lambdas gotten in step 1, and ...
0
votes
0answers
33 views

How dose dropout affect Weights and Bias?

I applied dropout in my network , and it worked , but i can't interpret dropout effects on weight and bias, to be more specific , i can't interpret why appling droput and not applying dropout have a ...
6
votes
1answer
143 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
2
votes
0answers
28 views

$L^2$ Regularization and Hessian Matrix [duplicate]

In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
5
votes
1answer
76 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
2
votes
0answers
30 views

In regularised regression, why not penalise higher degree coefficients more?

for the case of fitting a polynomial to data ($y = a_{1}x+a_{2}x^{2}+...+a_{n}x^{n}$) with regularisation of the coefficients, I'm wondering why are all of the coefficients usually penalised by the ...
2
votes
1answer
132 views

Why increasing lambda parameter in L2-regularization makes the co-efficient values converge to zero [duplicate]

Why increasing lambda parameter in L2-regularization makes the co-efficient values converge to zero? I have just tried to do the math, but it's a little bit rusted. Lets say that we have a simple ...
0
votes
0answers
17 views

Implicit regularization in Linear models

Regarding Linear Neural Networks models with unique finite root loss function, without an explicit regularization, I am struggling to prove that in the case of overparmeterized models (i.e. $N<d$), ...
0
votes
0answers
29 views

Selection criteria for penalty parameters in the ridge multinomial logit model

I appeal to you for the following doubt. I am adjusting a ridge multinomial logit model but I have problems in the criterion when choosing the lambda parameter that gives better results, besides ...
1
vote
0answers
14 views

A generalized LASSO constraint

I want to use LASSO in R but shrinking towards some fixed vector $A$, instead of shrinking towards 0. The desired L1 constraint, given coefficient vector $\beta$, is $||\beta-A||_{1} \leq k$, rather ...
0
votes
0answers
31 views

Regularization of linear regression problem [duplicate]

Consider a vector $a \in R^n$. I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where $f(x) = ||x-a||_{2}^2 + \lambda ||x|...
0
votes
0answers
28 views

SVM training yields too many (or no) support vectors

So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine. However, ...
0
votes
0answers
35 views

How is the generalization of LASSO called?

I know that ridge regression is a special case of Tykhonv regularization. In fact with Tykhonov one tries to minimize: $|| Ax - b ||^2 +|| \Gamma x ||^2$ If $\Gamma$ is the identity matrix scaled by ...
0
votes
1answer
50 views

Data augmentation methods for Raman Spectra

I'm building a CNN model based on Raman spectroscopy data and I wanted to experiment with data augmentation. What would be some reasonable techniques to try? I have found this paper which suggests ...
0
votes
0answers
36 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
93 views

Can one usefully specify a multilevel-model with a partially-nested, partially non-nested structure?

Background Gelman and Hill's Data Analysis Using Regression and Multilevel/Hierarchical Models includes an example in section 13.5 of how to model non-nested data. The second example in this section ...
2
votes
1answer
28 views

What value of alpha should I choose regularization

What value of alpha should I choose in glmnet? Should I use one which minimizes the cross-validation error, one which is one standard deviation above or below the one which gives the best error (like ...
0
votes
0answers
25 views

Adding regularization to an objective function when not using gradient descent

Using a simple example if I have a model: $$y = \beta_1 X_1 + \beta_2X_2 + {\rm error}$$ with cost function $${\rm Cost}= RSS + \alpha (\beta_1 + \beta_2)(\beta_1 + \beta_2)$$ If we were to use ...
0
votes
0answers
40 views

caret chooses non-optimal RMSE?

I run a linear regression via caret / glmnet method with "RMSE" as metric. In the final model, caret tells me which values of the tuning parameters alpha and lambda were selected to minimize RMSE. If ...
2
votes
0answers
231 views

Whether to use dropout vs. batch normalization vs L1/L2 loss for regularization

I am familiar with how dropout, batch normalization, and L1/L2 loss all work. However, I do not have an intuitive sense on when to use which. There are lots of discussion between dropout and batch ...
0
votes
1answer
30 views

How to regularize estimated probabilities in linear regression?

I estimate a linear probability model of the form: $ E[y\mid X] = X\beta $ where $y$ is a binary variable (hence, $E[y\mid X]= Pr(y=1\mid X)$) and $X$ a matrix coding a categorical variable by ...
4
votes
0answers
96 views

Do we have the actual theoretical study of L1/L2 regularization for Logistic regression?

It is very well known that L1 and L2 regularization can help in reducing the generalization error, and their effectiveness has been empirically demonstrated across a large set of machine learning ...
1
vote
1answer
850 views

Difference between kernel, bias, and activity regulizers in Keras

I've read this post, but I wanted more clarification for a broader question. In Keras, there are now three types of regularizers for a layer: kernel_regularizer, <...
1
vote
0answers
23 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
0
votes
1answer
169 views

Total variation regularization in deep learning

For current deep learning models, we can find basically two kinds of regularization on: Activation Weights The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization ...