Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
1
vote
1answer
42 views

What is the correct way to report the results of a penalized (Ridge, LASSO, ElasticNet) logistic regression model?

I was wondering what would be a sensible way to report the results of a penalized logistic regression model in a scientific article (in terms of coefficients, metrics, diagnostics, hyperparameters, ...
0
votes
0answers
34 views

Diagnostic checks and optimization of (ElasticNet) penalized logistic regression models (using glmnet and caret)

I was recently advised to use a penalized logistic regression model to better grasp what drivers influence my outcome (i.e. the eradication success/failure of an invasive plant species after a ...
0
votes
0answers
8 views

Extracting the value of the constraint given a known lambda parameter in LASSO

The LASSO problem is expressed as \begin{equation} \hat{\theta}\in \arg\min_{\theta\in\mathbb{R}^d} \left\{\frac{1}{2n}\lVert y-X\theta\rVert_2^2+\lambda_n\lVert\theta\rVert_1\right\} \end{equation} ...
0
votes
0answers
15 views

How can shrinkage methods help you improve model fit? [duplicate]

How can shrinkage methods help you improve model fit?
3
votes
3answers
85 views

Correcting for the under-estimation of regularized coefficients

L1 regularisation works by applying a penalty that reduces the magnitude of the coefficients. Some coefficients are driven to zero while the others are reduced but remain non-zero. The remaining non-...
0
votes
0answers
15 views

When can we not drop an independent variable that has a high VIF?

In a linear regression context, and we observe that some independent variable can be approximately written as a linear combination of a set of other independent variables (e.g., with $R^2 > 0.95 \...
0
votes
0answers
19 views

Logistic regression with regularization and Netwon-Raphson

Given the Optical Recog- nition of Handwritten Digits Data Set data set, which consists of 1000x64 training data sets and a 1000x1 vector label, how can you run a logistic regression with ...
0
votes
0answers
15 views

How to show that the lasso estimator is bounded in probability

The lasso estimator is defined as $$ \hat{\boldsymbol{\beta}}_{n}=\text{argmin}_{\boldsymbol{\beta}}\frac{1}{n}\left\Vert \mathbf{y}-\mathbf{X}\boldsymbol{\beta}\right\Vert_2^2 +\frac{\lambda_{n}}{n}\...
2
votes
1answer
80 views

Is there any justification for not standardizing predictors on disparate scales when using Lasso/Ridge?

I've looked at some Kaggle notebooks lately of people using Lasso/Ridge for linear regression. The majority that I've seen don't seem to standardize the predictors before they fit Lasso/Ridge even ...
1
vote
1answer
41 views

l1-regularization of network weights not going to zero

I'm working on an autoencoder that transforms a very high dimensional space (~10,000 inputs) through two hidden layers of 256 nodes each. (I settled on these values given the reconstruction error but ...
1
vote
1answer
26 views

How to approach this Lasso problem on the credit dataset

I am using the credit dataset to predict the balance of an individual. The specification of the model is non-linear however, in the following form: $$y_i = \beta_0 + \sum^J_{j=1}\beta_jx_{ij} + \sum^...
1
vote
1answer
34 views

How to optimise penalty parameter in ridge regression using AIC

So I know for a ridge regression model, we need to find an optimal $\lambda$ value. I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that ...
4
votes
1answer
75 views

What does a $L^{0.5}$ norm for regression regularization look like?

I am wondering how I can visualize or understand the $L^{0.5}$ norm in regression settings. In other words, the loss function is $$ \sum_{i=1}^{n}\left(Y_i-\sum_{j=1}^{p} X_{ij}\beta_j\right)^2 + \...
1
vote
0answers
11 views

How do the training and cross validation mean squared error curves behave as a function of $\lambda$?

I am currently looking into methods of choosing optimal tuning parameter $\lambda$ for ridge regression. I think that for the cross-validation the MSE should be relatively high for $\lambda=0$. Then I ...
0
votes
0answers
15 views

Choosing the right weight-decay value for a shrinked dataset

I used to train my deep networks on a dataset of 500k images with weight_decay=0.0001. The dataset consists of 500 classes, each having 1000 training images. I ...
0
votes
0answers
19 views

R: Plot of the relationship between lambda values and coefficients in ridge regression

I'm using the code below to plot the relationship between the lambda values used of ridge regression and the coefficients: ...
0
votes
0answers
12 views

R: Interpreting optimal values of lambda for ridge regression the using both default and pre-defined results

I am trying to create a ridge regression model and I am currently estimating the optimal value for lambda. ...
4
votes
0answers
54 views

What does the term “regularization” refer to specifically?

Initially I thought that "regularization" referred to specific methods to reduce overfitting by putting a penalizer term in the cost function that uses a norm eg. L1, L2 norm. But recently I'...
0
votes
0answers
13 views

Zero Mean assumption in theory but not in practice?

The paper "Network Inference via the Time-Varying Graphical Lasso" by David Hallac, Youngsuk Park, Stephen Boyd, Jure Leskovec shows how a (time varying) covariance matrix can be shrunk in ...
0
votes
0answers
16 views

How does the hyperparameter lambda affect L2 norm [duplicate]

Let's say I have L2 regularization in ridge regression: How would I go about giving a formal mathematical proof that I know that the larger the lambda the smaller the L2 norm. But, I don't know how ...
0
votes
0answers
16 views

Grouped-LASSO glmnet for logistic regression

I am trying to fit a logistic regression using grouped-lasso. The data contains some continuous predictors and categorical predictors with different levels. I did a quick internet search to see how to ...
0
votes
0answers
12 views

Are hyperparameters chosen from cross-validation slightly biased towards greater regularization?

I intend to fit a single model to the entire dataset after selecting hyperparameters by k-fold cross-validation. So on each round of training, my model is fit to $\frac{k-1}{k}n$ of my dataset, and ...
1
vote
0answers
17 views

Explain L1 vs. L2 regularization difference using the scientific mindset [duplicate]

I need to have job interview soon, one of the questions may be L1. vs L2 regularization. Yann LeCun explained best to my knowledge the difference between L1 and L2 regularization. L1 or Lasso: ...
0
votes
0answers
15 views

When using regularization wouldnt it make all parameters very small? [duplicate]

In regularization, we add square of thetas multiplied by lambda(excluding theta_0). The value of lambda is high because values of theta should be close to zero to neglect the value of its associated ...
2
votes
0answers
33 views

Effect of L2 regularization on Linear Regression

I am starting with L2 regularization on linear regression. Case without regularization: objective function $(Xw-y)^T(Xw-y)$ parameters vector $w= (X^TX)^{-1}X^Ty.$ Case with regularization: ...
1
vote
1answer
65 views

Intuitive explanation behind the statistical interpretation of regularization

I understand that using regularization is equivalent to finding a MAP estimate. I am wondering why using a Gaussian prior (for example) is better at preventing overfitting than using the uniform prior....
0
votes
0answers
15 views

How can I get p-values from a regularized GLMM?

I have a dataset containing information about patients in a hospital, with the following variables: Status for a certain disease (binary outcome) Hundreds of continuous biomarkers A few variables for ...
0
votes
2answers
38 views

Why is the optimal C chosen by GridSearchCV so small?

I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem. The issue I'm having is that when I run the code the optimal C is chosen to be ridiculously small (~e-18) so ...
1
vote
0answers
24 views

Time-series modeling w/ shrinkage for time-series model residuals?

First to fix notation, let $x_k$, $k=1,2,\dots$ denote a times series. Let $\hat{x}_{k+1} = f(x_k,x_{k-1},\dots)$ denote a prediction of $x_{k+1}$. Let $\epsilon_k = x_k-\hat{x}_{k}$ denote the ...
0
votes
0answers
14 views

On the test of significance under penalized likelihood estimation

I start using brglm2 package to implement logistic regression under a perfect separation problem. Is there any way to test the significance of the parameters using ...
0
votes
0answers
46 views

Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
0
votes
0answers
10 views

On the residual of the logistic regression fitted with penalized log-likelihood

I'm using logistf package in Rto to handle the problem of the complete separation for logistic regression but I cannot find the ...
1
vote
0answers
14 views

Can regularisation reduce the accuracy in the validation test?

I am constructing a CNN neural network with TensorFlow. I have run two versions of the CNN, one of them without regularization and the other with a kernel regularizer $L^2$ in each convolutional layer....
2
votes
0answers
16 views

How to interpret thresholding functions plots

In the paper The adaptive lasso and its oracle properties, page 4, they show several images like these ones: The caption of the image only states "Figure 1. Plot of Thresholding Functions". ...
1
vote
1answer
27 views

How to create a loss function to reflect different costs of misclassification errors?

I am currently reading 'Pattern Recognition and Machine Learning' and came across an example of a confusion matrix comparing the actual vs predicted number of people with or without cancer. The ...
1
vote
0answers
23 views

Can dropout negatively impact performance by increasing repetition?

Dropout is the idea that you can drop, i.e set to zero, some of the nodes in a computational neural network. The goal of this is to increase regularization by preventing the model from relying too ...
1
vote
0answers
13 views

How can the RMSE of penalized regression be lower than linear regression?

It appears to me that linear regression is choosing a $\hat{\textbf{w}}$ in $$E[Y] = \textbf{w}^T \textbf{x} \quad \quad (1)$$ by minimizing the least squares criterion $$\sum_{i=1}^N (y_n - \textbf{w}...
1
vote
0answers
15 views

Weight regularisation in CNN

I am trying to understand the concept of weight regularisation in CNN. I know that in dense layer with weight $w$ it corresponds to finding: $$ \mathbf{w}^{*}=\underset{\mathbf{w}}{\arg \operatorname{...
1
vote
0answers
21 views

What good are confidence intervals after regularization?

Suppose I run a regularized regression model such as Lasso. For simplicity let's say it's a linear model. After using cross-validation to find the $\lambda$ parameter, the model is refit (without ...
0
votes
0answers
22 views

What if zero mean assumption is relased in graphical LASSO?

I am working on a graphical LASSO (GLASSO) shrinkage of the variance-covariance matrix of financial log-returns data for 10 years. The objective of the graphical LASSO is: $$\ell(0,\Sigma) = {-\text{...
1
vote
0answers
18 views

What kind of regularization can I use for CNN aside from L1/L2/Dropout? [closed]

I am building a CNN to estimate a sequence of pitches existed in a song with this architecture: ...
23
votes
2answers
1k views

Should One Hot Encoding or Dummy Variables Be Used With Ridge Regression?

For a regression problem in which the predictor is a single categorical variable with $q$ categories, Ridge regression can be considered the Best Linear Unbiased Predictor (BLUP) for the mixed model $$...
0
votes
0answers
17 views

In what ways are complex machine learning algorithms different from simpler machine learning approaches? How to select your model

In what ways are complex machine learning algorithms (e.g, random forests or support vector machines) different from simpler machine learning approaches (e.g., LASSO or ridge regression)? Is this ...
0
votes
0answers
44 views

Penalizing autoregressive models (with R packages)

Consider the class of linear or quasi-linear models that include an autoregressive term, such as autoregressive distributed lag models, ARIMA models, VAR and VECM models, and so forth. In general the ...
0
votes
0answers
16 views

Implementing L1 and L2 penalty in sequential coordinate descent least squares

I'm trying to implement L1 and L2 regularization in a fast RcppArmadillo function for non-negative least squares. The function below is adapted from the NNLM R package, receives an initial value for <...
0
votes
0answers
9 views

When do you add probes to regularized regression?

When using regularized regression, my process is to first get a list of $\lambda$, use cross-validation to find the optimal $\lambda^*$, then refit the model on the full data using this $\lambda^*$. I ...
2
votes
0answers
27 views

Can you overcome overfitting by having a large enough sample size?

Can having a large sample size whereby n>>p (more specifically having about 6000 observations for each predictor) help overcome overfitting issues in a predictive model? If so, would this be as ...
0
votes
0answers
25 views

Linear regression: Best way of shrinking parameter estimates to mitigate multiple testing issues

Suppose I calibrate $N$ linear regression equations $(j=1,...,N)$, where $N$ is large (at least 100). The $j$th equation is: $\boldsymbol{Y}_{j}=\boldsymbol{X}\boldsymbol{\beta }_{j}+\boldsymbol{\...
7
votes
1answer
916 views

Why regularization parameter called as lambda in theory and alpha in python?

I was learning about regularization and came across the term called regularization parameter. I see that it is called lambda in ...
1
vote
0answers
105 views

L1 vs L2 norm - Circle and Diamond [duplicate]

I am new to ML and recently came across the L1 and L2 norm. The tutorials that I read here and here show some circle and diamond ...

1
2 3 4 5
24