Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
1 vote
1 answer
23 views

Stepwise model selection by AIC

I am learning about performing stepwise model selection by AIC and having some questions: What is the regularization parameter for step-AIC? In what way is forward step-AIC an evolution of univariate ...
user avatar
  • 41
3 votes
1 answer
82 views

In elastic net regularisation, will dividing the OLS term the number of observations cause misleading results when cross-validating?

Two formulations of the elastic net regression function Consider sklearn's implementation of elastic net regularisation (Wikipedia link). From the docs, it works by ...
user avatar
  • 31
1 vote
1 answer
15 views

Do I need to normalize data before applying L1, L2 norm in ANN

I wish to train the ANN and use regularizers to avoid overfitting. I need some suggestions, is it mandatory to normalize the data before using L1, L2 regularizers. I would highly appreciate if you can ...
user avatar
  • 141
0 votes
0 answers
11 views

Apply Shrinkage Coefficient to Exponentially Weighted Moving Covariance Matrices

Title says it all. I'm working on doing volatility forecasting and like the approach of exponentially weighted moving covariance matrices, but I also know that applying shrinkage coefficients further ...
user avatar
0 votes
0 answers
9 views

The use of the index argument in glmmlasso in R with interaction terms

I am using the glmmLasso package in R for variable selection with repeated measurements. I have 14 variables + interactions of each of these 14 variables with age that I want to use the selection for. ...
user avatar
0 votes
0 answers
11 views

Shrinkage / L1 regularization as a loss term versus a constraint (post-process step) with momentum optimizers

I have a complex model with very non-linear operations (divisions, exponentials, matrix inversions, square roots, Cholesky decompositions, etc...) for which I want to optimize the parameters. However, ...
user avatar
0 votes
0 answers
17 views

Detailed comparison of two methods for obtaining the ridge regression solution

I have come across two different ways of obtaining the ridge regression solution, which are as follows: Method1:-(obtained from here) $RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
user avatar
  • 187
3 votes
0 answers
33 views

Ridge Regression Alpha/Lambda: Basic Characteristics?

I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (...
user avatar
0 votes
0 answers
11 views

How to correct for an underestimating (yet consistent) nonparametric step function estimator of survival?

I am working with a specific type of data where the non-parametric step-function estimator of the survival function is underestimating the true survival function for small sample sizes yet I can prove ...
user avatar
0 votes
1 answer
51 views

Forcing covariates to always be part of a Lasso model

I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
user avatar
  • 1
2 votes
1 answer
34 views

Can we regularize/penalize the KNN model?

While reading through KNN in detail, I was checking if there is a way to improve/penalize KNN? I didn't find any concrete/easily understandable solutions.
user avatar
  • 21
0 votes
0 answers
11 views

Can a basis expansion guarantee no worse performance than original features?

Consider the typical learning problem where given inputs $x_i \in \mathbb{R}^p$ and targets $y_i \in \mathbb{R}$ for $i = 1, \dots, n$ we would like to learn some function $f$ such that $L(f(x_i), y_i)...
user avatar
  • 436
0 votes
0 answers
12 views

Would l-1 regularization with kernel trick induce sparsity on feature map's features?

Would l-1 regularization with kernel trick induce sparsity on the infinite dimensional feature map's features in the case of gaussian kernel?
user avatar
  • 224
0 votes
0 answers
8 views

Would logistic regression/support vector-machine with l-2 regularization and early stopping regularization cause underfitting?

Would early stopping regularization combined with l-2 regularization or in logistic regression/support vector machine cause underfitting? Does a kernel-trick affect what combination of regularization ...
user avatar
  • 224
0 votes
0 answers
13 views

Beta distribution equivalence with two redondant parameters [duplicate]

context In Factor graphs on discrete variables, the parameters are contained in factors associated each with a subset of the random variables in the system. Each factor provides a different positive ...
user avatar
13 votes
3 answers
2k views

If I use a regularization (e.g. L2) can I not apply early stopping?

I've seen that early stopping is a form of regularization that limits the movement of the parameters $\theta$ in a similar way that L2 Regularization penalizes the movement of $\theta$ to be closer to ...
user avatar
  • 287
2 votes
0 answers
28 views

Is regularization in Keras equivalent to a standard Ridge or Lasso problem?

With the python package Keras, you can use $\ell_2$ or $\ell_1$ regularization but you have to use the option on each layer. But I definitely cannot tell if using ...
user avatar
0 votes
0 answers
46 views

Deciding between the L1 and L2 penalty for a Sklearn Logistic Classifier

I have a classification problem with the following example independent features: recommendations comment_count comment. 0.663 . 0.382 'yes', 'trump' The dependent variable is whether the comment is ...
user avatar
  • 1
0 votes
0 answers
14 views

Regressor-based L2 penalty [duplicate]

I'm working on a multiple regression problem where I have reasons to believe some (if not all) of the regressors have been cherry picked/data mined to a varying degree. My hypotheses are that there's ...
user avatar
  • 749
2 votes
0 answers
25 views

Min-max scaling vs standardizing in LASSO

I know that it is recommended to have features on the same scale for LASSO, such that the scale does not affect the penalty. However, does it matter whether or not features are scaled using $\frac{x-\...
user avatar
  • 21
2 votes
2 answers
83 views

Why use regularization instead of feature selection for logistic regression? [duplicate]

For a non-linearly separable problem, when there are enough features, we can make the data linearly separable. It seems to me that for logistic regression, the reason of overfitting is always ...
user avatar
0 votes
0 answers
21 views

Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
user avatar
  • 11
1 vote
0 answers
19 views

Machine learning regularization parameter lambda proof [closed]

Consider the regularized empirical risk minimization problem L(x) + λ * r(x), where L(x) is the empirical risk, r(x) is the regularizer, and lambda is the regularization hyper-parameter. I have 2 ...
user avatar
  • 11
0 votes
0 answers
11 views

Wasserstein Adversarial Imitation Learning

in the paper, its mentioned that the authors used L2 regularization during training as shown below. −1/4ε(r(y) − r(x) − d(x, y))2 but it is not clear to me how to implement it, any hints?.
user avatar
  • 1
1 vote
0 answers
23 views

Single/ multiple imputation in post-selection/-regularization context

Context of problem: In some situations researchers face high-dimensional problems with $p > n$, where $p$ is the number of covariates to be considered in a regression model and $n$ is the sample ...
user avatar
  • 327
0 votes
0 answers
45 views

Regularization Terms in MLE

Can you add regularization terms to any likelihood function you're trying to maximize? (e.g. L2/Tikhonov, Lasso terms) I'm used to seeing this done with simple quadratic loss functions (e.g. for ...
user avatar
  • 117
4 votes
2 answers
236 views

Relationship between laplace and l1 regularization

It is well known that an L1 regularized linear regression is equivalent to a regression with a Laplace prior on the distribution of the coefficients. This is explained here: https://bjlkeng.github.io/...
user avatar
1 vote
0 answers
32 views

Penalized Regression for Predicting Rare Events

I have a data set where my binary dependent variable is rarely equals one (about 0.3 % of all observations). My goal is to predict the dependent variable based on a few variables and a constant term. ...
user avatar
0 votes
0 answers
67 views

How is type.measure="class" defined for cv.glmnet when using family = "multinomial"

cv.glmnet allows you to run cross validation for a multinomial logistic model to determine the value of lambda to use for LASSO. There are different ways that the ...
user avatar
2 votes
1 answer
74 views

Optimizing logistic regression with a custom penalty using gradient descent

I'm trying to fit a logistic regression model on a certain dataset. I want to ensure the learned model is smooth, that is samples which belong to the same cluster/group according to a prior knowledge/...
user avatar
  • 131
3 votes
2 answers
117 views

How to calculate the degrees of freedom for L1 and L2 regularised GLMs?

My goal is to calculate various information criteria for generalised linear models (e.g., the AIC). To do this, we need to calculate the effective degrees of freedom of the trained model. In an ...
user avatar
  • 131
6 votes
2 answers
558 views

Lasso coefficient for some features is higher than Linear Regression Coefficient

I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, ...
user avatar
0 votes
1 answer
216 views

Is Ridge more robust than Lasso on feature selection?

My goal is to identify the best n-feature linear model, i.e. pick the model with only n-feature from total N features (n < N) and lowest Mean-Squared-Error (MSE). The experiment is on the Lasso and ...
user avatar
  • 63
0 votes
0 answers
40 views

How to derive a regularized machine learning objective function with the maximum a posteriori for random features?

My question is at the end of the post. I tried to give as much information as I can to clarify my understanding and to point out as precisely as possible where I am stuck. Independent variables or ...
user avatar
0 votes
1 answer
28 views

Why is Scikit's Support Vector Classifier returning support vectors with decision scores outside [-1,1]? Is this a mistake?

I'm currently playing around with support vector machines in Scikit Learn and I've come across some unusual behaviour. For a basic simulated dataset, I've trained an SVC estimator (with linear kernel),...
user avatar
0 votes
0 answers
30 views

Regularization: How do you penalize weights of some exact value?

Assume we have a loss function of the form: L(f(X; theta),T) where X is the input dataset and T is the target dataset? Then you would update the the paramters by doing p = p - p.grad where p.grad is ...
user avatar
3 votes
1 answer
46 views

Shrinkage for the Cox model: how to select intensity $\lambda$?

The 2nd edition of James et al. "An Introduction to Statistical Learning" (2021) contains a new chapter on survival analysis and censored data (Chapter 11). Section 11.6 discusses shrinkage ...
user avatar
0 votes
0 answers
14 views

Multiple imputation in setting of penalized Cox regression - model convergence

I am fitting Cox regression models (the outcome is time-to-death) along with multiple imputation using PROC MI and the MCMC statement to address missing covariate data in SAS (assuming multivariate ...
user avatar
  • 1
5 votes
1 answer
99 views

When not to use the elastic net penalty in regression?

There is no shortage of resources on the advantages of using an elastic net penalty over L1 and L2 penalties (LASSO and ridge respectively). The following questions pertain to situations where ...
user avatar
  • 2,049
0 votes
1 answer
70 views

Examples in Machine Learning with Non-Differentiable Objective Functions

I was reading the following lecture notes on Gradient Descent and came across the following note: Supposedly, there are some instances in machine learning where the objective function is non-...
user avatar
  • 5,718
0 votes
0 answers
56 views

Can we call "Regularization" as "Constrained Optimization"?

I have the following question on "Regularization vs. Constrained Optimization" : In the context of statistical modelling, we are often taught about "Regularization" as a method of ...
user avatar
  • 5,718
1 vote
0 answers
48 views

How to insert a constraint on coefficients of logistic regression model?

I want to build a logistic regression model on my data, which contains three continuous predictors and a logical response. I need to constrain the regression coefficients to be not less than 0 and sum ...
user avatar
0 votes
0 answers
14 views

Disconnected subnetworks for uncorrelated estimators

When we have a data sample and we want to estimate two uncorrelated parameters of a conditional distribution, we can do this by just training two neural networks, one for each parameter. We could also ...
user avatar
  • 105
0 votes
0 answers
43 views

Logistic regression - are interaction terms redundant vs original features if using L1 penalization for feature selection?

I am running lasso/elastic regression for feature selection in a logistic classifier. I have two continuous features, and was wondering if it would be redundant to include an interaction term or other ...
user avatar
0 votes
0 answers
21 views

Optimality conditions for the LASSO

In this paper, on page 1122, it states that the optimality conditions for the LASSO give $\hat{\beta} = n_{\lambda}(\hat{\beta} - X^T(X\hat{\beta} - y))$, where $n_\lambda$ is the soft-thresholding ...
user avatar
  • 1
0 votes
0 answers
68 views

Prediction of a variable that lies within the interval $[0,1]$ with masses at the ends

I have a data set on kilometers travelled by households and the associated means of transport and now want to predict a means of transport's share in households' total kilometres travelled based on ...
user avatar
  • 1
0 votes
0 answers
28 views

In ridge regression, Why choose regression vector which has a minimum length?

As I reading a thesis named 'Ridge Regression: Biased Estimation for Nonorthogonal Problem' written by Hoerl and Kennard, I was struck by the below problem. Let $\boldsymbol{B}$ be any estimate of the ...
user avatar
0 votes
0 answers
19 views

Bayesian Approach for Underdetermined Datasets

If Bayesian Linear Regression with Gaussian prior produces L2 norm and Laplacian Prior produces L1 norm, is it fair to say that handling of underdetermined data sets (where number of columns > ...
user avatar
0 votes
1 answer
42 views

Can one use NRI and IDI in regularized cox-regression?

I have a dataset with 1500 patients for which I want to predict the outcome of death. I wanted to utilize multivariate cox-regression in a model containing biomarkers and other covariates. I was told ...
user avatar
  • 37
0 votes
0 answers
52 views

History of Regularization and Shrinkage [duplicate]

Can anyone recommend any research papers where the undesirable effects of overfitting on statistical models were first observed? In the context of regression, at what point did researchers begin to ...
user avatar
  • 5,718

1
2 3 4 5
26