# Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

1,401 questions
Filter by
Sorted by
Tagged with
11 views

### For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
12 views

### Intuition for how individual coefficients change with increasing regularization penalties

I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
43 views

1 vote
29 views

### What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as $$\arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2$$ This is quite similar to the objective function for plain OLS, except without the $W$ term: ...
1 vote
75 views

### Robust way to add predictors to existing linear model

I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
1 vote
7 views

### Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?

If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
1 vote
20 views

### Understanding application Lasso and Ridge Regression

Currently reading up on Ridge and Lasso regression, have some questions to clarify. Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5) ...
17 views

### weird lasso prediction when using lambda 1se

I have performed a leave-one out cross-validated prediction using a lasso regression (with both lambda min and lambda 1se). My sample size is 52 and I have a bit more than 20 predictors. While lambda ...
38 views

### Is $\ell_1$ regularization not compatible with SVM?

In the notes of Andrew Ng's CS229 Machine Learning course, it is mentioned: The $\ell_2$ norm regularization is much more commonly used with kernel methods because $\ell_1$ regularization is ...
8 views

### How can we compare elastic-net regularisation v.s equentially applying lasso and ridge regression?

If we use the below formulation of elastic-net penalisation: $$\beta^{EN} = {\rm argmin} \left( \lVert X-X\beta \rVert_2 +\lambda ( \alpha \lVert x \rVert_1 +(1-\alpha)\lVert x \rVert_2\right)$$ I ...
15 views

### Interpreting coefficients in Linear regression with categorical variables and one hot encoding (drop first)

I am doing multiple linear regression where my independent variables are a mix of categorical and numerical variables. Obviously I need to one-hot-encode the categorical variables, and I need to "...
23 views

### Can it Work to Run PCA only on a Subset of Highly Correlated Predictors prior to Regularization? (Cox)

I'm running Cox-LASSO (using glmnet as explained by Tay et al) on about 50 variables with about 300 observations. The variables fit into different categories like &...
14 views

### Proximal operator of Adaptive Elastic Net

I would like to learn how to find the proximal operator of the Adaptive Elastic Net, from DOI: 10.1214/08-AOS625 "ON THE ADAPTIVE ELASTIC-NET WITH A DIVERGING NUMBER OF PARAMETERS" by HUI ...
24 views

### Generalized cross-validation (GCV) with nonzero prior mean

I came across this concept of GCV optimization (new to me) for tuning hyperparameters in a model, as an alternative to maximizing the maginal likelihood (MML) of the output, which is what I am used to ...
4 views

### How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

I am trying to train a text classifier with open-source data to generalize on the real user traffic (henceforth "real data"). However, even though I have many annotated open-source data, I ...
57 views

### What exactly is the KKT check and what is the point of it?

In the paper for strong screening rules for the lasso (link), the following screening algorithm is proposed (start of chapter 7): Let $S(\lambda)$ be the strong rule set. Then the following strategy ...
194 views

### In a Ridge regression, why do i get a stronger shrinkage when i remove some coefficients from the penalization term?

I cannot understand why in a ridge regression if I remove some coefficients from the penalty term I have a stronger shrinkage of the remaining coefficients that are included in the penalty term. From ...
17 views

### Estimators that are superefficient on a dense set

In Chapter 8 of van der Vaart's Asymptotic Statistics, it is shown that (under weak regularity conditions) an estimator can be "superefficient" on at most a set of Lebesgue measure zero (...
10 views

### Betas from precision matrix?

This is really 2 questions. I would like to do a linear multi-regression on a large quantity of data, so large that I cannot really store it (it’s about 1e10 observations across 2500 features). Hence ...
30 views

### Is R-squared valid for regularized linear models?

I found that there has been extensive discussion on the invalidity of R-squared for nonlinear models according to its original definition based on the following mathematical analysis,. The variance in ...
For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^... 0 votes 0 answers 17 views ### Divide weight decay values by the learning rate values in a grid search? I have come across a paper where the authors do a grid search over hyperparameters. In particular, they tested different learning rates and weight decays. One thing that caught my attention was that ... 0 votes 1 answer 29 views ### Which standardization for lasso and dependent variable What kind of standardization of the predictor matrix is needed for lasso? let y be the response variable is it (x-mean(x)/sd(x)) OR (x-mean(x)/sd(y))? Also does lasso require you to standardize the ... 0 votes 0 answers 32 views ### Lasso coefficients and penalization impact What is the correct interpretation of the coefficients from a lasso regression if you standardize ((x-mean(x))/sd(x) the predictor variables? If you observer 4.2 for variable A and -1.2 for variable B ... 0 votes 0 answers 30 views ### Can LASSO still perform regularization on summarized data? Currently, we are trying to predict future revenue from existing users. We use the revenue collected after 14 days of membership to predict 3 year membership. We train the model and make predictions ... 6 votes 1 answer 146 views ### Can MCP or SCAD penalized regression be rephrased as a Bayesian regression with a particular prior on the coefficients? So given that the coefficients of a ridge regression with a squared L2 norm penalty corresponds to the maximum a posteriori (MAP) estimate of a Bayesian regression with Gaussian priors on the ... 0 votes 0 answers 10 views ### Why does the variance term in the VICREG paper encourage the variance to be equal to γ? In section 4.1 of the VICreg paper the authors describe the variance regularization term as the following hinge loss. $$v(Z) = \frac{1}{d}\sum_{j=1}^{d}max(0, \gamma - S(z^j, \epsilon ))$$ Z being ... 0 votes 0 answers 23 views ### When using early stopping for image classification with graph convolutional networks, which value for minimal improvement and patience is recommended? I'm using early stopping for avoiding overfitting in a task of image classification with a graph convolutional network. I'm trying to adopt a patience of 5 and a minimal improvement of 0.001. Is this ... 0 votes 0 answers 44 views ### How do you deal with a high Feature/Sample ratio? I am working on a machine learning task with a small dataset of 130 samples, each with 66 features. When I try to fit a model to this data, I encounter issues with either overfitting or underfitting. ... 3 votes 0 answers 50 views ### How to show the existence of global minimizer of Lasso type of objective function? Suppose the objective function to be minimized is $$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$ where$\theta$is the independent variable which is feasible in$\mathbb{R}^p$... 2 votes 0 answers 65 views ### Why do we need$\gamma>2$in SCAD penalty? The SCAD penalty$p(x | \lambda, \gamma)$from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ... 1 vote 3 answers 79 views ### How best to regularize high-cardinality fixed effects? Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ... 0 votes 0 answers 7 views ### Selecting sparse number of predictors each derived from a non-sparse b-spline in time I have observations that I would like to model as responses at stations to slowly time-varying inputs at 80 candidate locations, few (perhaps 5-10?) of which are active and significant. I have ... 0 votes 0 answers 36 views ### How to perform ridge penalisation on an interaction between discrete and continuous variables in mgcv? Using the paraPen term and gamma to control how much penalisation, what is the correct structure of penalty matrix input for <... 0 votes 0 answers 56 views ### Do we need to apply "multi-testing" corrections for the p-values in a regularized model? Say we are fitting a penalized model, such as a linear regression with lasso regularization. We expect to obtain a model with the most significant covariables. The method starts with many covariables ... 0 votes 0 answers 55 views ### How to choose the regularization term and why is the ridge regularization term W^2 in linear regression? I am working on implementing linear regression with regularization, and I'm trying to understand how to decide the appropriate regularization term to use. Specifically, I would like to know why in ... 0 votes 0 answers 21 views ### What is the basis generally used in software for the natural cubic splines? Of course there are many natural cubic spline bases, but reading through documentation (mgcv, splines in R and patsy, scipy.interpolate and csaps in Python) sheds no light on which ones are used in ... 0 votes 0 answers 31 views ### Why does scaling of one predictor influence the coefficient estimates of other predictors in ridge regression? In Introduction to Statistical Learning it is written The standard least squares coefficient estimates discussed in Chapter 3 are scale equivariant: multiplying$X_j$by a constant$c$simply leads ... 0 votes 0 answers 35 views ### Why the posterior predictive distribution is called the 'gold standard' in the Dropout paper? The dropout paper states that: With unlimited computation, the best way to regularize a fixed-sized model is to average the predictions of all possible settings of the parameters, weighting each ... 3 votes 1 answer 128 views ### How to handle BatchNorm in the last layers of a deep learning model? I am creating a neural network using batchnorm as a regularization method to enable deep models and prevent overfitting. I understand that batchnorming supresses the internal covariance shift ... 0 votes 1 answer 35 views ### From a computational perspective, how does the lasso regression shrink coefficients to 0? I understand the analytic proof that lasso regularisation tends to shrink coefficients to zero. However, from a practical standpoint, most of those methods are combined with gradient optimisation (... 1 vote 1 answer 115 views ### Why does a Logistic Regression with 'ElasticNet' penalty overfit? I have the problem that my logistic regression model overfits. even though I use the combined L1 and L2 penalty ('elastic net'). I have a data set with 496 features and 186 samples and want to predict ... 3 votes 1 answer 119 views ### Parameter distribution of$\theta$from a rectangular matrix multiplication$C\theta$I am struggeling to see where this problem fits - i.e. what topics this problem relates to, so I am not able to find the right literature. I want to use some particular information as a prior to a ... 2 votes 0 answers 73 views ### How to compute P-splines penalties with Harville's algorithm? I am trying to understand how to actually implement the Harville-Fellner-Schall algorithm for the estimation of the P-splines penalties presented in Appendix E (Sec. E.2) of Marx and Eilers . Let ... 1 vote 0 answers 26 views ### Fused lasso for image denonising For a given data$y_{i}$, with$i=1, \dots, n$, we consider the following signal approximation: $$\hat{y} = \arg \min_{w}\sum_{i=1}^{n}(y_{i}-w_{i})^{2} + \lambda \sum_{(i,j)\in E}|w_{i} - w_{j}|,$$ ... 1 vote 0 answers 8 views ### What are the relations between overfitting and susceptibility to adversarial perturbations in classification? For simplicity I am asking this question for a binary classification problem. In the non-robust case, for a dataset$(x_i, f(x_i))$,$i=1,\dots N$with$x_i$drawn iid from a distribution$D$, we ... 1 vote 0 answers 35 views ### L1 vs L2 variance? [closed] Which regularisation method L2 or L1 gives a lower variance?$ f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta ||^2 \rightarrow L2  f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta || \rightarrow L1 ... 