Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Regularization / weight decay [closed]

Why do we prefer smaller weights instead of larger weights for the same loss value?
0
votes
0answers
16 views

Regularization vs Hyper Parameter optimisation

I was hoping to get some informed opinion of the following problem. Currently we are fitting a timeseries problem by doing hyperparameter optimisation over multiple models. For instance comparing say ...
0
votes
1answer
36 views

Is there a way to ensure LASSO regularisation retains certain features in R? [closed]

I am creating a predictive model with a large number of features in R, but would like to prevent basic demographic features from being selected out of the model via LASSO regularisation. Is there a ...
3
votes
1answer
37 views

What prior would lead to $\ell_\infty$ regularization of model weights?

Gaussian prior on weights of a GLM lead to Ridge / $\ell_2$ squared regularization. Laplace prior leads to $\ell_1$ regularization Question What prior would lead to $\ell_\infty$ regularization ?
1
vote
1answer
45 views

Random-walk prior with ridge-like regularizarion?

I am working with a model that contains a large number of coefficients, arranged in an ordered vector $\beta_1, \dots, \, \beta_N $. I have some prior knowledge that could be used to improve the ...
1
vote
1answer
63 views

How should regularization parameters scale with data size?

I am choosing parameter vectors $\beta$ and $\nu$ to minimize an expression of the form: $$-\log{L(Y;X\beta,\nu)}+\frac{1}{2}\lambda {(\beta - \beta_0 )}^{\top} {(\beta - \beta_0 )}$$ where $\lambda$...
1
vote
0answers
26 views

Getting ElasticNet weight estimates

Regarding Zou and Hastie (2005), they formulate an augmented problem with $(X^*, y^*, \lambda_1, \lambda_2)$, which they they say can be used in a lasso-type problem where $\alpha = \frac{\lambda_1}{\...
1
vote
0answers
11 views

Elastic Net: diverging number of parameters

I am reading the paper On The Adaptive Elastic-Net With a Diverging Number of Parameters by Zou and Zhang (2009). I found it while I was researching the lasso and elastic net in general and I am ...
1
vote
0answers
16 views

How can robust regression be used with penalty functions for sparse solutions?

I don't seem to see robust regression used with penalty functions for variable selection, either L1 or Huber can't use mathematica's Fit at the same time
0
votes
0answers
24 views

Are my regularization results telling me that my model isn't overfitted?

In fitting a neural network, initial validation and learning curves seemed to indicate my model was overfitted. After trying a few options to decrease overfitting that had no impact, I'm questioning ...
5
votes
2answers
167 views

L1-regularization enforces sparsity for convex functions

I have a convex function $f \colon \mathbb{R}^n \to \mathbb{R}$ that I minimize using L1-regularization: $\DeclareMathOperator*{\argmin}{arg\,min}$ $$ x^*=\argmin_x f(x) + \lambda ||x||_1 $$ Can I ...
1
vote
0answers
13 views

Glmnet R - can't modify fdev parameter when lower = 0 [closed]

I want to solve the following optimisation problem $\hat{\beta} = \arg \min_{\beta \geq 0} \| y- A\beta\|_2^2 + \lambda \|\beta\|_1$ For that, I am using glmnet package (cv.glmnet for finding $\...
1
vote
1answer
52 views

Statistical library for orthogonal distance regression with a ridge penalty?

There are many libraries in R and python for doing orthogonal distance regression and for doing ridge regression separately. Is there one for doing them at the same time?
0
votes
0answers
27 views

Backpropagation of hinge loss with L2 regularization

I am trying to figure out the derivatives of multi-class linear model with hinge loss and L2 regularization: $L(x,y) = max(0, 1 - (\sum_{i}{x_{i}} W_{i, t} + b_{t}) + (\sum_{i}{x_{i}} W_{i, k} + b {k}...
1
vote
0answers
14 views

When regularizing based on an informative prior, how to give model a little more freedom to partially reject regulariziation

I am new here I hope this question is appropriate. I am modelling a spatial domain, whereby I have repeated measures at n locations. I make a bayesian linear model at each n locations based on about ...
1
vote
1answer
44 views

What's the relationship between the regularization parameter lambda and the constrain parameter K

In regularized regression, for example the ridge regression, we have the Lagrange method, which adds lambda times the 2-norm of parameters to the loss function and minimizes this. On the other hand, ...
0
votes
1answer
24 views

Neural Network Architecture Search

I'm applying NN for regression purpose. The model has 30 Input nodes, 1 hidden layer and 1 output. In order to find the optimal architecture of the hidden layer, I've constructed a loop that: tests ...
0
votes
0answers
9 views

Variance of ridge regression estimator under perturbed data

In section 1.3 Ridge Regression as Perturbation of the notes the author comes up with the following ridge estimator $$ \widehat{\boldsymbol{\beta}}=\left[(\mathbf{X}+\mathbf{W})^{T}(\mathbf{X}+\mathbf{...
1
vote
1answer
54 views

How does size of training set affect the regularization parameter found by cross validation?

Is it true that: Suppose you perform linear regression with $L_2$ regularization and use cross-validation to select the value of the regularization parameter $λ$ on two datasets drawn from the same ...
0
votes
0answers
24 views

Relation between regularization and Lagrange multipliers

This is sort of a follow-up to What is the connection between regularization and the method of lagrange multipliers ? that I thought deserved a new question. Consider an optimization problem of the ...
0
votes
0answers
10 views

Which covariate is significant in a regularized GLM?

Let’s say we have 10 covariates ($X_1$, $X_2$, …, $X_{10}$) and we want to test how important $X_2$ is in predicting our neuron’s response ($Y$). What we do now is to compare two models (nested models)...
3
votes
1answer
63 views

Regularization via model averaging?

Say you have the model $$ \Phi^{-1} \left(y\right)=\beta_0 + \beta_1 x$$ I am interested in adding some regularization, specifically concerning the parameter $\beta_1$, to introduce some "skepticism" ...
0
votes
1answer
21 views

l2 lambdas in Keras.regularizers [closed]

Is the value supplied to the shrinkage regularizers (l1 and l2) in Keras the inverse of the lambda coefficient? e.g. ...
0
votes
0answers
161 views

Lasso regression doesn't converge in case of zero Y-vector

I try to use lasso regression to solve linear problem with big amount of equations (~10 000). Everything worked fine, but I noticed that if in Y-vector all elements are equal, "fit" function hang for ...
0
votes
0answers
11 views

How to do cross validation in ODE models with more predicted than measured time courses?

I have an ODE model of biochemical reactions with 37 state variables and 88 strictly positive parameters. Unfortunately, I can only expect to get time course measurements of about 10 state variables (...
1
vote
2answers
101 views

What are the benefits of sparse representations and sparse parameters?

What are their benefits? I know sparse parameters are a different story than sparse representations, but I want to know how each of these can benefit us and which one is more important than the other ...
0
votes
0answers
8 views

Does SV-Regression incorporate something like the regression-effect from linear regression?

I am currently trying to understand the support vector regression and I have the following question about the hard margin SVR: Recall that the hard margin SVR solves the following optimization ...
0
votes
0answers
10 views

How to enforce smoothness in guided image filtering techniques ? Any preferable model?

Which one (or more) of these three minimization models is the appropriate way to enforce smoothness in guided filtering framework ? \begin{eqnarray} %\begin{aligned} & \sum\limits_{q \in {N}(p)} {\...
0
votes
0answers
19 views

What's the rationale behind multiple response LASSO?

I understand that, with LASSO, the regularization term puts a constraint on the complexity of our regression model. Usually, for prediction applications, regularization makes the model perform better ...
1
vote
1answer
30 views

how does noise-to-signal ratio effect the data splitting in training,validation and test sets

How will we divide our data set in training, validation and test set for model selection and model assessment? As in the book "Elements of statistical learning" page no.222 the author has mentioned ...
0
votes
1answer
27 views

Why Ridge regression add penalty to RSS instead of RSS/N (mean squared error)?

From Elements of Statistical Learning page 64, $loss = \sum (y- X\beta)^2 + \lambda \sum \beta^2$ The formula is not invariant to amount of data. If the data point doubles, $\lambda$ should be ...
3
votes
1answer
42 views

Why glmnet 's $\lambda$ value is so small? Does it strictly implement the loss function under the hood?

I am running a glmnet fit with 1200000 samples. According to the glmnet doc, $\lambda$ value is the coefficient controlling how much the regularization term contributes to the total loss function. ...
0
votes
0answers
42 views

Estimated coefficients after SAS glmselect with Lasso

I did run a lasso logistic regression with SAS glmselect (Y=1 for event and Y=-1 for non event). My syntax is something like: ...
1
vote
3answers
82 views

Regularization in Linear Regression

Let us assume a 1-dimensional regression problem (with one input variable). We are given a set of data points that are NOT collinear and our objective is to fit a straight line that best fits the ...
0
votes
0answers
43 views

LASSO Regression with noise

I know LASSO regression is useful to exclude redundant features, so can it be useful when you have noisy data? I explain better with this example: Suppose I generated a data set using an equation (e....
0
votes
0answers
16 views

What is considered as a good fit using the fraction deviance?

I am using the fraction deviance given by Hastie et. al (2015), $\mathcal{D}_{\lambda}^{2}=\frac{\mathcal{D}\mathrm{ev}_{\mathrm{null}}-\mathcal{D}\mathrm{ev}_{\lambda}}{\mathcal{D}\mathrm{ev}_{\...
0
votes
0answers
20 views

Regularize Regression with ARIMA errors in R

I am fitting regression with ARIMA errors in R. The xreg variables could be correlated with each other. Plus, I may be over-fitting my models. So, to handle both multicolinearity and over-fitting ...
0
votes
0answers
16 views

Complex output layer regularization implementation

I’m building a NN model using keras, and I wish to impose a constraint on it that doesn’t (directly) have to do with the weights. Would be very grateful for some help / points me towards some relevant ...
0
votes
0answers
15 views

Label smoothing for sequential image classification

My data are images of a car moving in a virtual environment, each example is classified as "left", "right", "straight", depending on the steering direction. I have a class imbalance, most of the ...
1
vote
0answers
50 views

Using validation data after early stopping

A common technique to do early stopping is to split the data to 3 parts: train, validation and test, and train on train set. After each epoch (or every K epochs) of training we check the loss of the ...
1
vote
1answer
75 views

Why standardization of design matrix $X$ with factor $\frac{1}{n}$ instead of $\frac{1}{n-1}$ in lasso/glmnet?

I'm a little bit puzzled by the default standardization of the lasso/elastic net/ridge regression algorithms implemented in the (great!) glmnet package. In most other applications, people would ...
0
votes
1answer
197 views

L2 Regularization in CatBoost

I am studying the CatBoost paper https://arxiv.org/pdf/1706.09516.pdf (particularly Function BuildTree in page 16), and noticed that it did not mention regularization. In particular, split selection ...
1
vote
1answer
59 views

Orthonormal regularizer to encourage diverse or non-redundant model parameters in neural networks

I was recently reading the paper Nian, F., Chen, X., Yang, S., & Lv, G. (2019). Facial Attribute Recognition With Feature Decoupling and Graph Convolutional Networks. IEEE Access, 7, 85500-85512....
0
votes
0answers
30 views

Random Slopes and Starting Parameters with GLMMLASSO

I am using glmmlasso in a simulation study. I want to decrease the time it takes to select the tuning parameter, lambda, by using the technique described in this answer: https://stats.stackexchange....
0
votes
0answers
11 views

What is the effective difference between PCA/SVD feature selection as input to logistic regression and Lasso regularization? [duplicate]

I have a problem with where the number of features (around 10k) is almost of the same order as the number of records in my data (around 100k). I'm using this data in a supervised classification task ...
0
votes
0answers
20 views

Can a classifier A get better result than classifier B when learning from the output of B?

I had the following problem recently: I tried to reverse engineer a classifier $C_1$. $C_1$ is an unknown, already in production classifier which I can't access. I can only access the result on past ...
0
votes
1answer
54 views

Determining Intercept for Regularized Logistic Regression

Going off of the standard set up, we have $N$ observations and $P$ predictors stored in the data matrix $\mathbf{X} = \{ x_{i,j} \}$ for $i = 1, \ldots, N$ and $j = 1, \ldots, P$. The response is ...
2
votes
1answer
165 views

L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....
1
vote
1answer
42 views

Measure of the coefficients variability for Regularised Regression models

I am working with Regression models. My idea is to measure the variability of the coefficients of some Regression models. I used LOOCV split for the training and testing my dataset. The ...
0
votes
0answers
15 views

Number of Variables in Elastic Net

I have a data set with 1000 observations and 150 independent variables. When I apply elastic net, I end up with 100 variables. I wonder if I need to do any additional feature selection or if I can use ...