Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
1,401
questions
3
votes
0
answers
11
views
For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?
A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
0
votes
0
answers
12
views
Intuition for how individual coefficients change with increasing regularization penalties
I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
2
votes
1
answer
43
views
Positive Semidefinite Kernel in RKHS
The following shows part of the page 170 of The Element of Statistical Learning that I want to make clear.
The solution can be characterized in two equivalent ways
$$\min_{c_j}\sum_{i=1}^N(y_i - \...
- 187
3
votes
1
answer
53
views
Is my regularized logistic regression model overfit?
I have a dataset with the following characteristics:
moderate sample size (~300 samples)
moderate class imbalance (~20% positives)
high-dimensional (the number of independent variables, again ~300, ...
- 41
1
vote
1
answer
67
views
Regularization Problem and Reproducing Kernel Hilbert Space
The following shows part of the page 169 of The Element of Statistical Learning that I want to make clear.
We have
$$\min_{f \in \mathcal H_K}[\sum_{i = 1}^NL(y_i, f(x_i)) + \lambda\Vert f\Vert_{\...
- 187
1
vote
1
answer
29
views
What is the objective function for weighted lasso & ridge?
For weighted OLS, the objective function can be written as
$$
\arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2
$$
This is quite similar to the objective function for plain OLS, except without the $W$ term:
...
- 1,127
1
vote
1
answer
75
views
Robust way to add predictors to existing linear model
I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
- 11
1
vote
0
answers
7
views
Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?
If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
- 11
1
vote
0
answers
20
views
Understanding application Lasso and Ridge Regression
Currently reading up on Ridge and Lasso regression, have some questions to clarify.
Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5)
...
- 11
0
votes
0
answers
17
views
weird lasso prediction when using lambda 1se
I have performed a leave-one out cross-validated prediction using a lasso regression (with both lambda min and lambda 1se). My sample size is 52 and I have a bit more than 20 predictors.
While lambda ...
- 1
0
votes
0
answers
38
views
Is $\ell_1$ regularization not compatible with SVM?
In the notes of Andrew Ng's CS229 Machine Learning course, it is mentioned:
The $\ell_2$ norm regularization is
much more commonly used with kernel methods because $\ell_1$ regularization is
...
0
votes
0
answers
8
views
How can we compare elastic-net regularisation v.s equentially applying lasso and ridge regression?
If we use the below formulation of elastic-net penalisation:
$$\beta^{EN} = {\rm argmin} \left( \lVert X-X\beta \rVert_2 +\lambda ( \alpha \lVert x \rVert_1 +(1-\alpha)\lVert x \rVert_2\right)$$
I ...
- 1
0
votes
0
answers
15
views
Interpreting coefficients in Linear regression with categorical variables and one hot encoding (drop first)
I am doing multiple linear regression where my independent variables are a mix of categorical and numerical variables. Obviously I need to one-hot-encode the categorical variables, and I need to "...
0
votes
0
answers
23
views
Can it Work to Run PCA only on a Subset of Highly Correlated Predictors prior to Regularization? (Cox)
I'm running Cox-LASSO (using glmnet as explained by Tay et al) on about 50 variables with about 300 observations. The variables fit into different categories like &...
- 11
0
votes
0
answers
14
views
Proximal operator of Adaptive Elastic Net
I would like to learn how to find the proximal operator of the Adaptive Elastic Net, from DOI: 10.1214/08-AOS625 "ON THE ADAPTIVE ELASTIC-NET WITH A DIVERGING NUMBER OF PARAMETERS" by HUI ...
- 35
0
votes
0
answers
24
views
Generalized cross-validation (GCV) with nonzero prior mean
I came across this concept of GCV optimization (new to me) for tuning hyperparameters in a model, as an alternative to maximizing the maginal likelihood (MML) of the output, which is what I am used to ...
0
votes
0
answers
4
views
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
I am trying to train a text classifier with open-source data to generalize on the real user traffic (henceforth "real data"). However, even though I have many annotated open-source data, I ...
- 217
0
votes
1
answer
57
views
What exactly is the KKT check and what is the point of it?
In the paper for strong screening rules for the lasso (link), the following screening algorithm is proposed (start of chapter 7):
Let $S(\lambda)$ be the strong rule set. Then the following strategy ...
- 55
0
votes
1
answer
194
views
In a Ridge regression, why do i get a stronger shrinkage when i remove some coefficients from the penalization term?
I cannot understand why in a ridge regression if I remove some coefficients from the penalty term I have a stronger shrinkage of the remaining coefficients that are included in the penalty term.
From ...
- 35
0
votes
0
answers
17
views
Estimators that are superefficient on a dense set
In Chapter 8 of van der Vaart's Asymptotic Statistics, it is shown that (under weak regularity conditions) an estimator can be "superefficient" on at most a set of Lebesgue measure zero (...
- 1,075
0
votes
0
answers
10
views
Betas from precision matrix?
This is really 2 questions.
I would like to do a linear multi-regression on a large quantity of data, so large that I cannot really store it (it’s about 1e10 observations across 2500 features). Hence ...
- 1
2
votes
1
answer
30
views
Is R-squared valid for regularized linear models?
I found that there has been extensive discussion on the invalidity of R-squared for nonlinear models according to its original definition based on the following mathematical analysis,.
The variance in ...
- 83
2
votes
1
answer
75
views
choice of m (order of derivative) for MGCV splines
I am working on a project that aims to estimate the association between a certain marker in blood and risk of an adverse cardiovascular outcome. Conventionally, a clinical threshold for the marker has ...
- 333
2
votes
0
answers
41
views
Distribution of Penalized Regression Coefficients
For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^...
- 3,400
0
votes
0
answers
17
views
Divide weight decay values by the learning rate values in a grid search?
I have come across a paper where the authors do a grid search over hyperparameters. In particular, they tested different learning rates and weight decays. One thing that caught my attention was that ...
0
votes
1
answer
29
views
Which standardization for lasso and dependent variable
What kind of standardization of the predictor matrix is needed for lasso?
let y be the response variable
is it (x-mean(x)/sd(x)) OR (x-mean(x)/sd(y))?
Also does lasso require you to standardize the ...
- 1
0
votes
0
answers
32
views
Lasso coefficients and penalization impact
What is the correct interpretation of the coefficients from a lasso regression if you standardize ((x-mean(x))/sd(x) the predictor variables?
If you observer 4.2 for variable A and -1.2 for variable B ...
- 1
0
votes
0
answers
30
views
Can LASSO still perform regularization on summarized data?
Currently, we are trying to predict future revenue from existing users. We use the revenue collected after 14 days of membership to predict 3 year membership. We train the model and make predictions ...
- 33.3k
6
votes
1
answer
146
views
Can MCP or SCAD penalized regression be rephrased as a Bayesian regression with a particular prior on the coefficients?
So given that the coefficients of a ridge regression with a squared L2 norm penalty corresponds to the maximum a posteriori (MAP) estimate of a Bayesian regression with Gaussian priors on the ...
- 3,040
0
votes
0
answers
10
views
Why does the variance term in the VICREG paper encourage the variance to be equal to γ?
In section 4.1 of the VICreg paper the authors describe the variance regularization term as the following hinge loss.
$$ v(Z) = \frac{1}{d}\sum_{j=1}^{d}max(0, \gamma - S(z^j, \epsilon )) $$
Z being ...
0
votes
0
answers
23
views
When using early stopping for image classification with graph convolutional networks, which value for minimal improvement and patience is recommended?
I'm using early stopping for avoiding overfitting in a task of image classification with a graph convolutional network.
I'm trying to adopt a patience of 5 and a minimal improvement of 0.001.
Is this ...
- 718
0
votes
0
answers
44
views
How do you deal with a high Feature/Sample ratio?
I am working on a machine learning task with a small dataset of 130 samples, each with 66 features. When I try to fit a model to this data, I encounter issues with either overfitting or underfitting.
...
- 389
3
votes
0
answers
50
views
How to show the existence of global minimizer of Lasso type of objective function?
Suppose the objective function to be minimized is
$$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$
where $\theta$ is the independent variable which is feasible in $\mathbb{R}^p$...
- 71
2
votes
0
answers
65
views
Why do we need $\gamma>2$ in SCAD penalty?
The SCAD penalty $p(x | \lambda, \gamma)$ from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ...
- 71
1
vote
3
answers
79
views
How best to regularize high-cardinality fixed effects?
Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ...
- 331
0
votes
0
answers
7
views
Selecting sparse number of predictors each derived from a non-sparse b-spline in time
I have observations that I would like to model as responses at stations to slowly time-varying inputs at 80 candidate locations, few (perhaps 5-10?) of which are active and significant. I have ...
- 201
0
votes
0
answers
36
views
How to perform ridge penalisation on an interaction between discrete and continuous variables in `mgcv`?
Using the paraPen term and gamma to control how much penalisation, what is the correct structure of penalty matrix input for <...
- 1
0
votes
0
answers
56
views
Do we need to apply "multi-testing" corrections for the p-values in a regularized model?
Say we are fitting a penalized model, such as a linear regression with lasso regularization. We expect to obtain a model with the most significant covariables.
The method starts with many covariables ...
- 1,064
0
votes
0
answers
55
views
How to choose the regularization term and why is the ridge regularization term W^2 in linear regression?
I am working on implementing linear regression with regularization, and I'm trying to understand how to decide the appropriate regularization term to use. Specifically, I would like to know why in ...
Vishal kumar Pal
0
votes
0
answers
21
views
What is the basis generally used in software for the natural cubic splines?
Of course there are many natural cubic spline bases, but reading through documentation (mgcv, splines in R and patsy, scipy.interpolate and csaps in Python) sheds no light on which ones are used in ...
- 163
0
votes
0
answers
31
views
Why does scaling of one predictor influence the coefficient estimates of other predictors in ridge regression?
In Introduction to Statistical Learning it is written
The standard least squares coefficient estimates discussed in Chapter 3
are scale equivariant: multiplying $X_j$ by a constant $c$ simply leads ...
- 59
0
votes
0
answers
35
views
Why the posterior predictive distribution is called the 'gold standard' in the Dropout paper?
The dropout paper states that:
With unlimited computation, the best way to regularize a fixed-sized model is to average the predictions of all possible settings of the parameters, weighting each ...
- 728
3
votes
1
answer
128
views
How to handle BatchNorm in the last layers of a deep learning model?
I am creating a neural network using batchnorm as a regularization method to enable deep models and prevent overfitting.
I understand that batchnorming supresses the internal covariance shift ...
- 281
0
votes
1
answer
35
views
From a computational perspective, how does the lasso regression shrink coefficients to 0?
I understand the analytic proof that lasso regularisation tends to shrink coefficients to zero.
However, from a practical standpoint, most of those methods are combined with gradient optimisation (...
1
vote
1
answer
115
views
Why does a Logistic Regression with 'ElasticNet' penalty overfit?
I have the problem that my logistic regression model overfits. even though I use the combined L1 and L2 penalty ('elastic net').
I have a data set with 496 features and 186 samples and want to predict ...
- 13
3
votes
1
answer
119
views
Parameter distribution of $\theta$ from a rectangular matrix multiplication $C\theta$
I am struggeling to see where this problem fits - i.e. what topics this problem relates to, so I am not able to find the right literature. I want to use some particular information as a prior to a ...
2
votes
0
answers
73
views
How to compute P-splines penalties with Harville's algorithm?
I am trying to understand how to actually implement the Harville-Fellner-Schall algorithm for the estimation of the P-splines penalties presented in Appendix E (Sec. E.2) of Marx and Eilers [1]. Let ...
- 385
1
vote
0
answers
26
views
Fused lasso for image denonising
For a given data $y_{i}$, with $i=1, \dots, n$, we consider the following signal approximation:
$$
\hat{y} = \arg \min_{w}\sum_{i=1}^{n}(y_{i}-w_{i})^{2} + \lambda \sum_{(i,j)\in E}|w_{i} - w_{j}|,
$$
...
- 426
1
vote
0
answers
8
views
What are the relations between overfitting and susceptibility to adversarial perturbations in classification?
For simplicity I am asking this question for a binary classification problem. In the non-robust case, for a dataset $(x_i, f(x_i))$, $i=1,\dots N$ with $x_i$ drawn iid from a distribution $D$, we ...
- 11
1
vote
0
answers
35
views
L1 vs L2 variance? [closed]
Which regularisation method L2 or L1 gives a lower variance?
$
f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta ||^2 \rightarrow L2
$
$
f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta || \rightarrow L1
...
user387816