Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
1
vote
1answer
39 views

Why standardization of design matrix $X$ with factor $\frac{1}{n}$ instead of $\frac{1}{n-1}$ in lasso/glmnet?

I'm a little bit puzzled by the default standardization of the lasso/elastic net/ridge regression algorithms implemented in the (great!) glmnet package. In most other applications, people would ...
0
votes
0answers
16 views

L2 Regularization in CatBoost

I am studying the CatBoost paper https://arxiv.org/pdf/1706.09516.pdf (particularly Function BuildTree in page 16), and noticed that it did not mention regularization. In particular, split selection ...
1
vote
1answer
33 views

Orthonormal regularizer to encourage diverse or non-redundant model parameters in neural networks

I was recently reading the paper Nian, F., Chen, X., Yang, S., & Lv, G. (2019). Facial Attribute Recognition With Feature Decoupling and Graph Convolutional Networks. IEEE Access, 7, 85500-85512....
0
votes
0answers
16 views

Random Slopes and Starting Parameters with GLMMLASSO

I am using glmmlasso in a simulation study. I want to decrease the time it takes to select the tuning parameter, lambda, by using the technique described in this answer: https://stats.stackexchange....
0
votes
0answers
9 views

What is the effective difference between PCA/SVD feature selection as input to logistic regression and Lasso regularization? [duplicate]

I have a problem with where the number of features (around 10k) is almost of the same order as the number of records in my data (around 100k). I'm using this data in a supervised classification task ...
0
votes
0answers
19 views

Can a classifier A get better result than classifier B when learning from the output of B?

I had the following problem recently: I tried to reverse engineer a classifier $C_1$. $C_1$ is an unknown, already in production classifier which I can't access. I can only access the result on past ...
0
votes
1answer
45 views

Determining Intercept for Regularized Logistic Regression

Going off of the standard set up, we have $N$ observations and $P$ predictors stored in the data matrix $\mathbf{X} = \{ x_{i,j} \}$ for $i = 1, \ldots, N$ and $j = 1, \ldots, P$. The response is ...
2
votes
1answer
60 views

L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....
1
vote
1answer
35 views

Measure of the coefficients variability for Regularised Regression models

I am working with Regression models. My idea is to measure the variability of the coefficients of some Regression models. I used LOOCV split for the training and testing my dataset. The ...
0
votes
0answers
13 views

Number of Variables in Elastic Net

I have a data set with 1000 observations and 150 independent variables. When I apply elastic net, I end up with 100 variables. I wonder if I need to do any additional feature selection or if I can use ...
0
votes
2answers
43 views

Why don't we use regularization on decision tree split?

I heard people ask which one is better: Linear regression with regularization or Random Forest. My question is why can't you use regularization with Random Forest? My understanding is that different ...
0
votes
0answers
15 views

Are residuals meaningful when using additive, robust, and regularization models on time series

If I am using additive models, regularization, or robust regression to model time series data, should I still check that the residuals have no autocorrelation and are stationary? I am under the ...
0
votes
0answers
23 views

Mathematical proof of how L1 and L2 regularization work [duplicate]

How do you mathematically prove that L1 regularization makes weights sparse but L2 regularization does not?
0
votes
0answers
14 views

intervention analysis with multivariate regression

Does intervention analysis make sense when one has multiple external regressors? How would one know if say, a level shift, is caused by a combination of external regressors or by a change in a law ...
0
votes
0answers
9 views

elastic net with random interventions

Since elastic net gets rid of regressors that do not correlate, would it make sense to randomly put in a bunch of interventions (pulses, level shifts, ect)? If the intervention is not needed, won't ...
2
votes
0answers
9 views

LeNet5 “symmetry breaking” step, is (was) it important?

Reading through Yann LeCun's original paper on LeNet5, I have come across something that I haven't seen before in convolutional neural network architectures. (Maybe that's just because I'm late to ...
2
votes
1answer
126 views

When to prefer PCA over regularization methods in regression?

When dealing with the curse of dimensionality, regularization methods seem to be clear in their intuition. All "regularization" methods can be seen as a "squeezing" of one's variables towards ...
0
votes
0answers
18 views

Debiased regularised regression (elastic net)

I have ~55,000 binary observations and 12 explanatory variables (EVs). I am looking to perform variable selection, followed by inference on the effect of the retained EVs on the binary outcome. Since ...
0
votes
1answer
35 views

Do I discard all my dependent variables as proved by chi-squared test of independence?

I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable ...
1
vote
0answers
6 views

Variation in accuracy of data splitting before and after data augmentation

How much accuracy of the system varied/changes between two cases Data augmentation before splitting Data augmentation after splitting, only on training data Is there any literature published?
0
votes
1answer
31 views

Data augmentation on entire dataset before splitting

If I apply rotation of 5 different angles and randomly cropp 10 different images from each rotated image and than divided into training testing and validation. Will it be totally incorrect evaluation ...
1
vote
2answers
60 views

In “A Topology Layer for Machine Learning,” are the topological priors learned by the network or imposed by humans?

In this paper by Gabrielsson, Nelson, et al. the authors "present a differentiable topology layer that can, among other things, construct a loss on the output of a deep generative network to ...
1
vote
0answers
14 views

Linear regression: How to demand similar MSE across different subgroups?

In typical least square regression, we want to minimize $||y-\hat{y}||$ where $\hat{y}=B*x$ I am now working on a car fleet management problem, $y$ can be split into several groups (in my case, ...
0
votes
0answers
54 views

Should we penalize dummy variables? [duplicate]

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
2
votes
0answers
10 views

Is there a formal relation between weight regularization and compression?

In my understanding, compression, strictly speaking, means that we diminish the amount of data required to describe something, such as a model. E.g. compressing an image file means to create a file ...
2
votes
1answer
37 views

How does the penalized form of RSS (residual sum of squares) work?

In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning. Note: I had used the ...
0
votes
1answer
39 views

Inconsistency between poisson and negativebinomial in glm

I am working with the negativebinomial distribution for GLM. I have done one test which is finding the poisson distribution results. Here is the first test: ...
0
votes
0answers
22 views

Terminology: “L1 regularization” even if I'm using mean instead of sum? [duplicate]

In my loss function I'm using the mean of the log-cosh error between the predictions and targets, as well as an additional regularization term that scales as the mean of the absolute value of another ...
1
vote
0answers
31 views

How does regularized regression overcome the p > n problem?

So, I understand why simple linear or logistic regression will have infinite solutions in this case (good answers here and here). But while LASSO will only select n features, Elastic net does not have ...
1
vote
1answer
31 views

Enforcing Dirac delta-like Activations on a Neural Network

I am working on a custom neural network model including convolutional and dense layers. I intend to enforce outputs a certain dense layer to approximate a Dirac delta function (or any localized pulse)....
15
votes
2answers
355 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
2
votes
1answer
18 views

Reference Request: Proof of Early Stopping Criterion

I am looking for a proof that "Validation-based early stopping" methods work but I have no idea where to start, as I am new to this field. Any recomendations of some rigerous papers that focus on ...
0
votes
0answers
22 views

Notation problem with sparse regularized correlation matrix

I am trying to apply a specific method to obtain a sparse correlation matrix $R$ from a regularized correlation matrix $\Sigma^{\delta}$, which was computed from $N$ observations of a multivariate ...
0
votes
0answers
27 views

Elastic Net - number of non-zero variables

I have a question regarding the interpretation of the trace of coefficients when running Elastic net with the package glmnet in R. This is the plot I obtain with alpha = 0.5 My understanding is that ...
0
votes
0answers
9 views

Supervised learning vs Regularity based learning

I have some confusion about regularity based learning and supervised learning. Are they in essence, not the same thing? We have some labelled data, and our algorithms are structured based on learning ...
0
votes
1answer
176 views

Why do we need regularized logistic regression?

We use regularized Linear Regression to prevent the model from overfitting (reduce model complexity). Does the same idea hold with regularized Logistic Regression? Is regularized Logistic ...
0
votes
1answer
27 views

Question about the location of regularization constant C in SVM

I've encountered very similiar but different functions in SVM optimization problem, the diffrence is in the location of regularization constant C. $\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2C} \left\...
8
votes
1answer
305 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
2
votes
1answer
91 views

James-Stein regularizing covariance like a mean

In James-Stein's estimator we have a $p$-dimensional random vector $X\sim N_{p}(\mu ,I)$ where $\mu \neq 0$ and the goal is to estimate the mean vector using the single ($n=1$) data vector $X$. The ...
3
votes
1answer
213 views

Why is lasso more robust to outliers compared to ridge?

In my attempt to reason about it intuitively I am concluding that ridge might be more robust to outliers. Following is my intuitive/lose reasoning : If there is an outlier then to match my ...
0
votes
1answer
20 views

How can I store information in a custom regularizer? [closed]

I'm trying to create a custom keras regularizer that uses the distance of the layer's weights from it's original weights, but what I used doesn't seem to work. I get a zero difference at all times. ...
3
votes
1answer
37 views

Should MLE estimation always be using penalizers?

I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some ...
1
vote
0answers
10 views

Does the expected cross validated error/NLL from regularisation have one minimum?

When trying to choose the value of the regularisation parameter(s) in lasso, ridge regression, or elasticnet, one generally computes the cross validated error or negative log-likelihood as explained ...
2
votes
1answer
40 views

How does L2 penalize large weights

The L2 regularization term is useful because it penalizes large weights over smaller weights which is good to prevent overfitting. I'm having a hard time understanding how exactly it does this. This ...
0
votes
0answers
19 views

Link between norm of weights/coefficients and smoothness

We often avoid overfitting by penalizing the norm of the weights/coefficients (in a classic Ridge or Lasso regression). I understand that we want smooth functions as they will be more likely to ...
0
votes
0answers
33 views

How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
0
votes
0answers
78 views

Lasso Logistic Regression in the presence of Class Imbalance

Since class imbalance only affects the estimate of the intercept in vanilla logistic regression, the orientation of the optimal separating hyperplane remains unchanged. However in $L_1$-regularized ...
3
votes
1answer
242 views

Does regularization penalize models that are simpler than needed?

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?
1
vote
1answer
21 views

Consisten Regularizer for Neural Network

In the book 'Pattern Recognition and Machine Learning' by Bishop (p.257 ff.) he considers a weight decay regularizer of the error function $$\hat E(w)=E(w)+\frac{\lambda}{2}w^tw$$ where $w$ is a ...
0
votes
0answers
14 views

Package to perform 2nd degree polynomial regression with L1 penalty for use of the 2nd degree

I'm trying to fit either a straight line or 2nd degree polynomial through many sets of points (2-dimensional data). I would much prefer a straight line over a polynomial, so am trying to penalize the ...