# Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

770 questions
Filter by
Sorted by
Tagged with
5 views

### Regularization / weight decay [closed]

Why do we prefer smaller weights instead of larger weights for the same loss value?
16 views

### Regularization vs Hyper Parameter optimisation

I was hoping to get some informed opinion of the following problem. Currently we are fitting a timeseries problem by doing hyperparameter optimisation over multiple models. For instance comparing say ...
36 views

### Is there a way to ensure LASSO regularisation retains certain features in R? [closed]

I am creating a predictive model with a large number of features in R, but would like to prevent basic demographic features from being selected out of the model via LASSO regularisation. Is there a ...
37 views

### What prior would lead to $\ell_\infty$ regularization of model weights?

Gaussian prior on weights of a GLM lead to Ridge / $\ell_2$ squared regularization. Laplace prior leads to $\ell_1$ regularization Question What prior would lead to $\ell_\infty$ regularization ?
45 views

### Random-walk prior with ridge-like regularizarion?

I am working with a model that contains a large number of coefficients, arranged in an ordered vector $\beta_1, \dots, \, \beta_N$. I have some prior knowledge that could be used to improve the ...
63 views

### How should regularization parameters scale with data size?

I am choosing parameter vectors $\beta$ and $\nu$ to minimize an expression of the form: $$-\log{L(Y;X\beta,\nu)}+\frac{1}{2}\lambda {(\beta - \beta_0 )}^{\top} {(\beta - \beta_0 )}$$ where $\lambda$...
26 views

52 views

### Statistical library for orthogonal distance regression with a ridge penalty?

There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
27 views

20 views

### Regularize Regression with ARIMA errors in R

I am fitting regression with ARIMA errors in R. The xreg variables could be correlated with each other. Plus, I may be over-fitting my models. So, to handle both multicolinearity and over-fitting ...
16 views

### Complex output layer regularization implementation

I’m building a NN model using keras, and I wish to impose a constraint on it that doesn’t (directly) have to do with the weights. Would be very grateful for some help / points me towards some relevant ...
15 views

### Label smoothing for sequential image classification

My data are images of a car moving in a virtual environment, each example is classified as "left", "right", "straight", depending on the steering direction. I have a class imbalance, most of the ...
50 views

### Using validation data after early stopping

A common technique to do early stopping is to split the data to 3 parts: train, validation and test, and train on train set. After each epoch (or every K epochs) of training we check the loss of the ...
75 views

### Why standardization of design matrix $X$ with factor $\frac{1}{n}$ instead of $\frac{1}{n-1}$ in lasso/glmnet?

I'm a little bit puzzled by the default standardization of the lasso/elastic net/ridge regression algorithms implemented in the (great!) glmnet package. In most other applications, people would ...
197 views

### L2 Regularization in CatBoost

I am studying the CatBoost paper https://arxiv.org/pdf/1706.09516.pdf (particularly Function BuildTree in page 16), and noticed that it did not mention regularization. In particular, split selection ...
59 views

### Orthonormal regularizer to encourage diverse or non-redundant model parameters in neural networks

I was recently reading the paper Nian, F., Chen, X., Yang, S., & Lv, G. (2019). Facial Attribute Recognition With Feature Decoupling and Graph Convolutional Networks. IEEE Access, 7, 85500-85512....
30 views

### Random Slopes and Starting Parameters with GLMMLASSO

I am using glmmlasso in a simulation study. I want to decrease the time it takes to select the tuning parameter, lambda, by using the technique described in this answer: https://stats.stackexchange....
11 views

### What is the effective difference between PCA/SVD feature selection as input to logistic regression and Lasso regularization? [duplicate]

I have a problem with where the number of features (around 10k) is almost of the same order as the number of records in my data (around 100k). I'm using this data in a supervised classification task ...
20 views

### Can a classifier A get better result than classifier B when learning from the output of B?

I had the following problem recently: I tried to reverse engineer a classifier $C_1$. $C_1$ is an unknown, already in production classifier which I can't access. I can only access the result on past ...
54 views

### Determining Intercept for Regularized Logistic Regression

Going off of the standard set up, we have $N$ observations and $P$ predictors stored in the data matrix $\mathbf{X} = \{ x_{i,j} \}$ for $i = 1, \ldots, N$ and $j = 1, \ldots, P$. The response is ...
165 views

### L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....