Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

l2 regularization: difference between activity_regularizer and kernel_regularizer [closed]

it's known that l2 regularization is equivalent to imposing gaussian priors on the weights but to apply these priors in keras should i use activity_regularizer or the kernel_regularizer ? thanks
  • 145
0 votes
0 answers
36 views

"Label smoothing" for sigmoid output [closed]

Is there something like label smoothing for sigmoid output? By that I mean creating more uncertainty during traning. I am thinking something like adding a noise to gold data? For example, with sigmoid ...
  • 1
1 vote
0 answers
11 views

Penalty Matrix in P-Splines with correlated data

Good evening, I'm having some trouble understanding how the penalty matrix in penalized splines is set up if the error term has a stationary process. Say I have $y=X\beta+ \epsilon$ with $\epsilon \...
0 votes
0 answers
5 views

Add a regularization term to the objective of a stable-baseline3 model [closed]

I'm using stable-baseline3's PPO implementation (see here) and wanted to play with the model a little bit further. More specifically, I wanted to add a ...
  • 115
0 votes
0 answers
8 views

Linear Model with fixed Weights and Terms / Ridge Regression / Regularization in R

I am working on setting up regression models for prediction in psychometrics and ran into challenges with cross validation. Essentially, I would like to have cross validated linear regression models ...
3 votes
3 answers
45 views

Is the magnitude coefficient vector in Ridge regression monotonic in lambda?

recently an interesting question came up and while I would have intuitively said it is not, other students have now made a compelling case (while not being sure themselves). For ridge (or l2 ...
  • 229
3 votes
1 answer
50 views

P-Splines with dependent data

I'm working through the generalized additive models book by Simon Wood and I've had a discussion with a friend of mine over how P-Splines estimation would work for dependent data. For independent data ...
1 vote
0 answers
35 views

Univariate Regression Coefficients and Multivariate Regression Coefficients

I got the following question: Suppose I have three variables, $x_1$, $x_2$ and $y$. We run univariate regression of $y$ on $x_1$ ($x_2$) with intercept and get the regression coefficients $\beta_1$ ($\...
  • 203
0 votes
1 answer
60 views

Geometrical interpretation of why can't ridge regression shrink coefficients to 0?

To explain the difference between Ridge and Lasso regression, following diagram is used as it is claimed that Ridge regression cannot shrink the regression coefficients to 0: But my question is, if ...
0 votes
0 answers
13 views

Interpretation/Intuition for L2 Regularization in Neural Networks [duplicate]

When we use L1-regularization in neural networks, it is pretty intuitive how the regularization will influence the learned weights. Namely, weights will not become needlessly large and unimportant ...
2 votes
1 answer
73 views

What does it say about the data if ridge regression is not reducing multicollinearity?

I am predicting the salary to be offered to a new candidate for which I am concentrating on just continuous (9 in number) variables. Variables are as attached. When I ran OLS the coefficient for total ...
4 votes
1 answer
309 views

Why we do train/val/test split in ML models, but not in regularized linear regression?

There is one thing in data science that I cannot understand. When we have algorithms like Random Forest, Gradient Boosting, Neural Networks, we split our data into three parts - train (to train our ...
  • 432
1 vote
1 answer
39 views

Subgradient for sparse-group lasso

Sparse-group lasso is defined as $$\frac{1}{2n}\left\|y-X\beta \right\| + (1-\alpha)\lambda\sum_{l=1}^m \sqrt{p_l}\left\|\beta^{(l)} \right\|_2 + \alpha \lambda \left\| \beta\right\|_1$$ In the SGL ...
  • 151
13 votes
3 answers
1k views

Why do we need regularization for linear least squares given that a line is the simplest model possible?

In linear least squares we are trying to fit a line to data. A line is the simplest model that can be fit to the data. How is it possible for a linear model to over-fit the data? In short why do we ...
  • 231
0 votes
0 answers
38 views

Why does the second early stopping method described in Deep Learning work?

Goodfellow et al.'s Deep Learning book describes a way of continuing training with the full dataset after early stopping: Another strategy for using all the data is to keep the parameters obtained ...
4 votes
1 answer
26 views

Should I use regularization in a price elasticity model?

I am building a price elasticity model using linear regression: log(demand) ~ 1 + log(price) + ... Does it make sense to use L1 and/or L2 regularization to prevent ...
  • 1,085
2 votes
2 answers
88 views

Can a regularization harm more than help in the situation of a huge over-fit?

I fit a regression model on a data set and get some in-sample RMSE. I wanted to know, how likely is that I get this good RMSE (or even better) under assumptions that there are no patterns in the data. ...
  • 355
0 votes
0 answers
19 views

Does Random Forest Regularization affect OOB error estimation?

I have a wide (~2000 columns), yet short (~70 rows) data set. I am using the R package ranger with regularization to determine if I can use these features to predict my binary outcome. I have noticed ...
8 votes
2 answers
185 views

Lasso regression Mathematical intuition

I do not fully understand lasso regression. I am able to code a LASSO regression model of the form $y = \beta_0 + \beta_1 x_1 + \ldots \beta_p x_p$ with sklearn, but I do not understand the ...
5 votes
2 answers
200 views

What is the meaning of the variance in a prior that represents L2 regularization?

Why is the L2 regularization equivalent to Gaussian prior? does a nice job in explaining the relationship between L2 regularization and Gaussian priors. One answer explains that the variance in the ...
0 votes
0 answers
23 views

Implementing Logistic Regression using Ridge in R but getting negative outputs?

I am trying to implement some logistic regression using ridge in R for predicting company bankrupcy using a subset of this data but despite my best efforts and various lecture notes/online tutorials, ...
  • 1
1 vote
0 answers
58 views

variance estimation for fused lasso

In the paper THE SOLUTION PATH OF THE GENERALIZED LASSO the authors derived degrees of freedom for the generalised lasso. Assume that $y\in\mathcal{N}(\mu,\Sigma)$, with $\mu\in\mathbb{R}^{n}$ and $\...
  • 376
0 votes
0 answers
20 views

Why do we take the maximum probability to define stable variables in stability selection?

In stability selection (link), we first calculate the selection probabilities, $\Pi_K^\lambda$. Then, using these probabilities, the stable variables are defined as $$S^\text{stable} = \{ k: \max(\...
  • 151
1 vote
0 answers
30 views

Is there an empirical rule for selecting the value of label smoothing?

I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
0 votes
0 answers
10 views

regularization for covariate reduction in a growth mixture model

I am planning to fit a growth mixture model and I have identified a very large number of covariates. These have been justified by consultation with expert groups (content experts in the field) and ...
  • 167
1 vote
0 answers
18 views

State-of-the-art techniques for regularizing Neural Networks?

For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's. Have any significant/strong competitors risen up since then?
  • 401
0 votes
0 answers
32 views

Imputation for Big Data [duplicate]

I have a dataset with 78 observations and 25000 variables.I am trying to apply logistic regression with shrinkage methods(penalties like SCAD,MCP,LASSO).The problem is that the commands ncvreg(ncvreg ...
  • 31
0 votes
0 answers
25 views

glmnet with weighted penalty

I need to fit a elastic net penalized logistic regression model in the form of Here W is a positive definite weight matrix. Since ...
  • 283
2 votes
0 answers
26 views

if my dataset is standardized but have outliers should I remove them and re-standardize? [closed]

I have a data set named Geographical Original of Music Data Set from the UCI repository. The data is given standardized but I think it has outliers and I do not know the best way to handle them. ...
  • 61
0 votes
0 answers
9 views

standard linear model vs linear model with an artificial restriction on the coefficient effects

Consider a standard linear model and one with an artificial restriction on the coefficient effects, where k is a fixed regularization parameter. Standard: $y = b_1 x_1 + b_2 x_2 + b_3 x_3$ Restricted: ...
  • 51
2 votes
0 answers
44 views

LASSO and duality theorem

I am confused with Lagrange duality theorem. Let us consider the problem $$ \hat{\beta} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} \left[\sum_{i=1}^{n}(y_{i} - \beta_{i})^{2} + \lambda \sum_{i=...
  • 73
1 vote
0 answers
78 views

Duality gap calculation in Scikit-learn implementation of Lasso

I am writing a custom variation of Lasso regression, using sklearn's Lasso implementation as a "source of inspiration". And I don't quite understand the very last line in the calculation of ...
0 votes
0 answers
25 views

Elastic net grouping property in logistic regression

The grouping property of the elastic net is a well-known property. The elastic net groups highly correlated variables together in its coefficient estimates. In Theorem 1 of the elastic net paper (here)...
  • 283
1 vote
0 answers
24 views

alternative solution to fussed lasso

The question is related to strange result from fused lasso estimator Let us consider fussed lasso estimator: $$ \hat{\beta}^{FL} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} [(y_{i} - \beta_{i})^{...
  • 73
4 votes
1 answer
78 views

strange result from fused lasso estimator

Let us consider the following estimator: $$ \hat{\beta}^{F} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} (y_{i} - \beta_{i})^{2} + \lambda_{1} \sum_{i=1}^{n-1}|\beta_{i} - \beta_{i+1}|, $$ which ...
  • 73
0 votes
0 answers
166 views

Impact of L1 and L2 regularisation with cross-entropy loss

When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
1 vote
2 answers
395 views

Why not perform weight decay on layernorm/embedding?

I am learning the code of minGPT. In the function, the author excluded layernorm and embedding layer from experiencing weight decay and I want to know the reasons. Besides, what about batchnorm?
1 vote
1 answer
34 views

Stepwise model selection by AIC

I am learning about performing stepwise model selection by AIC and having some questions: What is the regularization parameter for step-AIC? In what way is forward step-AIC an evolution of univariate ...
  • 51
3 votes
1 answer
117 views

In elastic net regularisation, will dividing the OLS term the number of observations cause misleading results when cross-validating?

Two formulations of the elastic net regression function Consider sklearn's implementation of elastic net regularisation (Wikipedia link). From the docs, it works by ...
  • 51
1 vote
1 answer
29 views

Do I need to normalize data before applying L1, L2 norm in ANN

I wish to train the ANN and use regularizers to avoid overfitting. I need some suggestions, is it mandatory to normalize the data before using L1, L2 regularizers. I would highly appreciate if you can ...
  • 141
0 votes
0 answers
25 views

Apply Shrinkage Coefficient to Exponentially Weighted Moving Covariance Matrices

Title says it all. I'm working on doing volatility forecasting and like the approach of exponentially weighted moving covariance matrices, but I also know that applying shrinkage coefficients further ...
0 votes
0 answers
37 views

The use of the index argument in glmmlasso in R with interaction terms

I am using the glmmLasso package in R for variable selection with repeated measurements. I have 14 variables + interactions of each of these 14 variables with age that I want to use the selection for. ...
0 votes
0 answers
37 views

Shrinkage / L1 regularization as a loss term versus a constraint (post-process step) with momentum optimizers

I have a complex model with very non-linear operations (divisions, exponentials, matrix inversions, square roots, Cholesky decompositions, etc...) for which I want to optimize the parameters. However, ...
0 votes
0 answers
20 views

Detailed comparison of two methods for obtaining the ridge regression solution

I have come across two different ways of obtaining the ridge regression solution, which are as follows: Method1:-(obtained from here) $RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
  • 187
3 votes
0 answers
100 views

Ridge Regression Alpha/Lambda: Basic Characteristics?

I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (...
0 votes
0 answers
16 views

How to correct for an underestimating (yet consistent) nonparametric step function estimator of survival?

I am working with a specific type of data where the non-parametric step-function estimator of the survival function is underestimating the true survival function for small sample sizes yet I can prove ...
0 votes
1 answer
66 views

Forcing covariates to always be part of a Lasso model

I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
  • 1
2 votes
1 answer
128 views

Can we regularize/penalize the KNN model?

While reading through KNN in detail, I was checking if there is a way to improve/penalize KNN? I didn't find any concrete/easily understandable solutions.
  • 21
0 votes
0 answers
13 views

Can a basis expansion guarantee no worse performance than original features?

Consider the typical learning problem where given inputs $x_i \in \mathbb{R}^p$ and targets $y_i \in \mathbb{R}$ for $i = 1, \dots, n$ we would like to learn some function $f$ such that $L(f(x_i), y_i)...
  • 456
0 votes
0 answers
16 views

Would l-1 regularization with kernel trick induce sparsity on feature map's features?

Would l-1 regularization with kernel trick induce sparsity on the infinite dimensional feature map's features in the case of gaussian kernel?

1
2 3 4 5
27