Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

1,274 questions
Filter by
Sorted by
Tagged with
7 views

Disconnected subnetworks for uncorrelated estimators

When we have a data sample and we want to estimate two uncorrelated parameters, we can do this by just training two neural networks, one for each parameter. We could also model this approach as ...
23 views

Logistic regression - are interaction terms redundant vs original features if using L1 penalization for feature selection?

I am running lasso/elastic regression for feature selection in a logistic classifier. I have two continuous features, and was wondering if it would be redundant to include an interaction term or other ...
11 views

Optimality conditions for the LASSO

In this paper, on page 1122, it states that the optimality conditions for the LASSO give $\hat{\beta} = n_{\lambda}(\hat{\beta} - X^T(X\hat{\beta} - y))$, where $n_\lambda$ is the soft-thresholding ...
40 views

Prediction of a variable that lies within the interval $[0,1]$ with masses at the ends

I have a data set on kilometers travelled by households and the associated means of transport and now want to predict a means of transport's share in households' total kilometres travelled based on ...
25 views

In ridge regression, Why choose regression vector which has a minimum length?

As I reading a thesis named 'Ridge Regression: Biased Estimation for Nonorthogonal Problem' written by Hoerl and Kennard, I was struck by the below problem. Let $\boldsymbol{B}$ be any estimate of the ...
16 views

Bayesian Approach for Underdetermined Datasets

If Bayesian Linear Regression with Gaussian prior produces L2 norm and Laplacian Prior produces L1 norm, is it fair to say that handling of underdetermined data sets (where number of columns > ...
21 views

Can one use NRI and IDI in regularized cox-regression?

I have a dataset with 1500 patients for which I want to predict the outcome of death. I wanted to utilize multivariate cox-regression in a model containing biomarkers and other covariates. I was told ...
40 views

History of Regularization and Shrinkage [duplicate]

Can anyone recommend any research papers where the undesirable effects of overfitting on statistical models were first observed? In the context of regression, at what point did researchers begin to ...
7 views

XGBClassifier regularization strategy help

I am trying to train XGBClassifier models on unbalanced data. I am performing a randomized grid search on a parameter space and using 'neg_log_loss' as the metric. The issue I'm having is that the ...
27 views

51 views

Are there any mathematical reasons that describe why "sparse models" are desirable?

I am interested in better learning about why Model Sparsity (i.e. Regularization) "works" - whether this is more due to mathematical principles or empirical results (on a case by case basis,...
13 views

R: Estimating LASSO and elastic net across multiple time-series CV approaches

This question is of a disapproved format and I am sure that I deserve to have some points taken away, but I hope someone will have pity on me and answer me anyway. I have been given access to a ...
35 views

9 views

Can regularization be used when your features >> number of observations to reduce the feature space?

I'm wondering if a reasonable way of reducing the feature space when p >> n is to simply use l1/l2 regularization. Will this work? Or can the model simply not be fit to begin with, and so the ...
76 views

L1 regularization for feature selection in neural net

In statistics, a lasso regression do some feature selection (or reduce the dimensionality of the problem). This is a very efficient technique as both the prediction and the feature selection use the ...
45 views

Lasso with Linear Constraints

Given $X \in \mathbb{R}^{n \times d}$ and $y \in \mathbb{R}^{n}$ I am trying to fit the linear regression model $$\min_{\beta \in \mathbb{R}^d} || y - X\beta||_2^2$$ under the constraints:  \beta \...
25 views

Data preprocessing before lasso regression

I am doing lasso for variable reduction from a bunch of 100 odd variables. Some numeric variable have extreme values. for e.g **count of rooms in house ** have values like 1,2,3,4,5,6,7,100. 100 is ...
78 views

Is there a way to write Elastic Net in expanded matrix form?

I am working through a regression problem for a matrix of data that isn't full rank and has more features than observations. For these reasons, I'd like to use elastic net because of its $L1$ and $L2$ ...
76 views

If LASSO is equivalent to Bayesian Regression with a Laplace (double exponential) prior, what would be the prior for non-negative LASSO? Exponential?

We know that the LASSO penalty is equivalent to Laplace prior. So what would be the corresponding prior for a non-negative LASSO? Is it exponential distribution? More generally, is it true that every ...