Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Disconnected subnetworks for uncorrelated estimators

When we have a data sample and we want to estimate two uncorrelated parameters, we can do this by just training two neural networks, one for each parameter. We could also model this approach as ...
0
votes
0answers
23 views

Logistic regression - are interaction terms redundant vs original features if using L1 penalization for feature selection?

I am running lasso/elastic regression for feature selection in a logistic classifier. I have two continuous features, and was wondering if it would be redundant to include an interaction term or other ...
0
votes
0answers
11 views

Optimality conditions for the LASSO

In this paper, on page 1122, it states that the optimality conditions for the LASSO give $\hat{\beta} = n_{\lambda}(\hat{\beta} - X^T(X\hat{\beta} - y))$, where $n_\lambda$ is the soft-thresholding ...
0
votes
0answers
40 views

Prediction of a variable that lies within the interval $[0,1]$ with masses at the ends

I have a data set on kilometers travelled by households and the associated means of transport and now want to predict a means of transport's share in households' total kilometres travelled based on ...
0
votes
0answers
25 views

In ridge regression, Why choose regression vector which has a minimum length?

As I reading a thesis named 'Ridge Regression: Biased Estimation for Nonorthogonal Problem' written by Hoerl and Kennard, I was struck by the below problem. Let $\boldsymbol{B}$ be any estimate of the ...
0
votes
0answers
16 views

Bayesian Approach for Underdetermined Datasets

If Bayesian Linear Regression with Gaussian prior produces L2 norm and Laplacian Prior produces L1 norm, is it fair to say that handling of underdetermined data sets (where number of columns > ...
0
votes
1answer
21 views

Can one use NRI and IDI in regularized cox-regression?

I have a dataset with 1500 patients for which I want to predict the outcome of death. I wanted to utilize multivariate cox-regression in a model containing biomarkers and other covariates. I was told ...
0
votes
0answers
40 views

History of Regularization and Shrinkage [duplicate]

Can anyone recommend any research papers where the undesirable effects of overfitting on statistical models were first observed? In the context of regression, at what point did researchers begin to ...
0
votes
0answers
7 views

XGBClassifier regularization strategy help

I am trying to train XGBClassifier models on unbalanced data. I am performing a randomized grid search on a parameter space and using 'neg_log_loss' as the metric. The issue I'm having is that the ...
0
votes
1answer
27 views

What kind of regularization is $\frac{1}{2} \sum_{i=1}^{D}\left(\frac{w_{i}^{2}}{\lambda_{i}}+\lambda_{i}\right)$?

$$\min _{\mathbf{w}} \min _{\lambda \geq 0}\left\{C \sum_{n=1}^{N}\left(t_{n}-\mathbf{w}^{T} \mathbf{x}_{n}\right)^{2}+\frac{1}{2} \sum_{i=1}^{D}\left(\frac{w_{i}^{2}}{\lambda_{i}}+\lambda_{i}\right)\...
2
votes
3answers
100 views

How does addition of a regularization term ensures that the matrix is nonsingular? ( least squares )

In Bishop's Pattern recognition book, in 3.1.2 Geometry of least squares section (page 143, last paragraph of section), it is stated that: In practice, a direct solution of the normal equations can ...
0
votes
0answers
5 views

Combining multiple datasets vs multiple models in high dimensions

This question is related to this one and this one, but I was wondering about this topic in general. Imagine a setting where multiple datasets, representing different measurements, have been gathered. ...
1
vote
1answer
57 views

"Milder" than a convolutional neural network: not forcing connections to be perfectly equal or exactly zero, but penalizing such behavior

Convolutional neural networks (CNNs) do regularization (of sorts) by forcing some weights to be dropped and others to be zero. Borrowing some drawings from another post of mine... Apply the filter to ...
1
vote
0answers
39 views

Studies on where to apply L2?

Are there any studies on where (and maybe how much) L2 to apply per parameter? E.g. in a more complex neural network, e.g. some encoder-decoder, with different components, from my own experience, just ...
1
vote
1answer
23 views

Decision trees and Feature Scaling for regularization

For all tree-based models like xgboost, lightgbm, random forest, they do not require feature scaling given the nature in which they compute their splits. However, when you perform regularization, one ...
3
votes
1answer
40 views

Regularization and Shrinkage : Theoretical Advantages vs. Empirical Advantages

I have the following question about the theoretical advantages vs. the empirical advantages of regularization (i.e. shrinkage). As far as I understand, this is the general idea behind regularization: ...
4
votes
1answer
72 views

How does early stopping act as a regulizer?

Reading the book "Deep Learning" by Goodfellow et al., in section 7.8 (for ref. Deep Learning - Chapter 7), I came across a demonstration of why early stopping can be interpreted as a ...
1
vote
1answer
53 views

Ridge Trace Plot - Interpretation

In my research, I aimed to perform a regression model with four predictors and one response variable. When I verified a high collinearity among the predictors, I was instructed to handle this problem ...
2
votes
2answers
48 views

Ridge regression with shrinkage towards nonzero matrix

Suppose I want to perform ridge-regularized linear regression, except that we shrink the coefficients to a nonzero matrix: $$ W^* = \arg\min_W \|Y - X W \|^2_2 + \lambda\|W-W_0\|^2_2. $$ However, I ...
0
votes
0answers
5 views

regularisation: choice of noise/ perturbation values for dataset with small number of observations

In a dataset $D$ of dimension $N \times d$ where the number of observation $N$ is small such that it leads to the covariance matrix being ill - conditioned, a workaround involves introducing ...
0
votes
0answers
27 views

Uniqueness of LASSO

I gather the LASSO estimator is unique when the columns of predictor matrix $X$ are in general position, as mentioned here. What is a specific example of predictor matrix $X$ that does not obey this ...
0
votes
0answers
17 views

Compatibility condition in LASSO

I am reading Statistics for High-Dimensional Data (Bühlmann and van de Geer). Chapter 6 discusses obtaining the oracle inequality in LASSO under the compatibility condition, a technical assumption ...
0
votes
1answer
34 views

How to obtain odds ratio (and 95% CI) from ridge regression model

I am currently working on a ridge logistic (predictive) model. I was able to complete most of the steps and obtain the coefficient but I keep getting an error message when it comes to the odds ratio &...
0
votes
0answers
21 views

shrinkage and glmnet Convergence for nth lambda value

I am using LASSO from glmnet-package to create predictions. Furthermore, I am using cv.glmnet-function to do 5-fold cross-validation to create Lasso.fit. This glmnet-object is then used in predict-...
0
votes
0answers
3 views

Household recruitment based on variable reduction technique

I want to recruit Households based on spend on liquor by identifying set of features which increases the average spend. Altogether there are 70 features having mixed type e.g. Type of house, ...
0
votes
0answers
28 views

Oracle inequality for LASSO

In chapter 6 of Statistics for High Dimensional Data (Peter BühlmannSara van de Geer), they focus on the normal linear regression model with fixed regressors and begin by stating OLS achieves a risk $\...
2
votes
1answer
26 views

Bayes prior in MAP estimation corresponding to $\ell^0$ penalization

I gather that in the context of penalized least squares, we can interpret a penalty term as corresponding to a prior $\pi(\beta)\propto \exp\{-\text{pen}\}.$ Is this also true for $\ell^0$ ...
3
votes
1answer
60 views

Bayesian priors associated with regularization penalties

I gather that adding a penalty term to (linear) least squares minimization typically corresponds with choosing some prior for Bayes estimation in the normal linear regression model. A couple questions ...
1
vote
1answer
14 views

How to use dev set to tune hyperparameters?

I have a quick question regarding train, dev, test set. We have a simple log linear model trained via gradient ascent on the log likelihood of the data combined with l2-regularizer. Our instructor ...
3
votes
1answer
45 views

Can the horseshoe prior be used with Beta(b, b) instead of Beta(.5, .5)?

The horseshoe prior for, say, an unknown mean $\mu$ is expressed $$ \begin{align*} \mu_i | \lambda_i, \tau &\sim N(0, \lambda_i^2 \tau^2) \\ \lambda_i &\sim C^+(0,1) \end{align*} $$ where the ...
0
votes
0answers
9 views

Adding New Regularization Terms to Closed Form LSE

I am trying to align the input space X to the output space Y by using least squares method in a closed form solution. To do that I use svd for finding the rotation matrix (W). And I find a solution ...
0
votes
1answer
26 views

How are the penalized splines defined here?

Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline $$ f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\...
3
votes
1answer
51 views

Are there any mathematical reasons that describe why "sparse models" are desirable?

I am interested in better learning about why Model Sparsity (i.e. Regularization) "works" - whether this is more due to mathematical principles or empirical results (on a case by case basis,...
0
votes
0answers
13 views

R: Estimating LASSO and elastic net across multiple time-series CV approaches

This question is of a disapproved format and I am sure that I deserve to have some points taken away, but I hope someone will have pity on me and answer me anyway. I have been given access to a ...
1
vote
0answers
35 views

Distribution of James-Stein estimator

For reference, I'm working with something like $${\mathbf Y} \sim N_d({\boldsymbol \mu}, \sigma^2 I)$$ We can estimate $\boldsymbol\mu$ using the JSE $$ \widehat{\boldsymbol \theta}_{JS} = \left( 1 - ...
2
votes
1answer
66 views

Variance of the ridge regression estimator

I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$): My main concern is that this derivation of ...
1
vote
1answer
31 views

Expected value of the ridge regression estimator

I am trying to understand this derivation: I think everything except the last equality is fairly simple, but I do not understand the last equality. Is there an error here? I appreciate any help.
0
votes
1answer
26 views

Proof of "Shrinkage" in Statistics

I have the following question regarding the Proof of "Shrinkage" in Statistics. Up to the present moment, I always thought that "Shrinkage" was a synonymous term with "...
0
votes
0answers
25 views

Scaling factor for binary and quadratic terms in penalized regression

The purpose of standardisation in penalised regression model are to create relatively "fair" condition between all covariates, where the value ranges are approximately the same. That is ...
2
votes
1answer
33 views

Bayesian Elastic Lasso

While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need ...
0
votes
0answers
31 views

Why does the ridge penalty shrink the singular values? [duplicate]

I am trying to understand the following analysis of ridge regression. I am new to SVD but I think I have a sufficient grasp on most of the content. There are two things I am struggling with. The ...
1
vote
0answers
30 views

What is the name for this error metric?

I am following a paper that says that a generalized learning method is optimized by minimizing a regularized error metric: $$ H(f_\mathcal{G}) = (1-\lambda)||y_i-f_\mathcal{G}(\mathbf{x_i})||^2 + \...
0
votes
0answers
9 views

Can regularization be used when your features >> number of observations to reduce the feature space?

I'm wondering if a reasonable way of reducing the feature space when p >> n is to simply use l1/l2 regularization. Will this work? Or can the model simply not be fit to begin with, and so the ...
2
votes
1answer
76 views

L1 regularization for feature selection in neural net

In statistics, a lasso regression do some feature selection (or reduce the dimensionality of the problem). This is a very efficient technique as both the prediction and the feature selection use the ...
0
votes
0answers
45 views

Lasso with Linear Constraints

Given $X \in \mathbb{R}^{n \times d}$ and $y \in \mathbb{R}^{n}$ I am trying to fit the linear regression model $$\min_{\beta \in \mathbb{R}^d} || y - X\beta||_2^2$$ under the constraints: $$ \beta \...
0
votes
0answers
25 views

Data preprocessing before lasso regression

I am doing lasso for variable reduction from a bunch of 100 odd variables. Some numeric variable have extreme values. for e.g **count of rooms in house ** have values like 1,2,3,4,5,6,7,100. 100 is ...
1
vote
1answer
78 views

Is there a way to write Elastic Net in expanded matrix form?

I am working through a regression problem for a matrix of data that isn't full rank and has more features than observations. For these reasons, I'd like to use elastic net because of its $L1$ and $L2$ ...
4
votes
1answer
76 views

If LASSO is equivalent to Bayesian Regression with a Laplace (double exponential) prior, what would be the prior for non-negative LASSO? Exponential?

We know that the LASSO penalty is equivalent to Laplace prior. So what would be the corresponding prior for a non-negative LASSO? Is it exponential distribution? More generally, is it true that every ...
0
votes
0answers
14 views

Sliding or extending window for shrinkage parameter estimation in time series cross-validation?

I do not understand something about the time-series cross-validation method put forward by Rob Hyndman's Forecasting: Principals and Practice, either 2nd or 3rd edition. I would think the whole point ...
2
votes
1answer
37 views

Why use dropout in feedforward?

Maybe I am just confused by what is the point of using dropout in the feed-forward? Wouldn't be better to forward the input with the whole network and then use the dropout only in the back-prop to ...

1
2 3 4 5
26