Questions tagged [shrinkage]

Shrinkage in statistics is a form of regularization consisting in changing parameter estimates to be "smaller" (closer to zero), or, more generally, "closer to each other".

Filter by
Sorted by
Tagged with
15
votes
2answers
346 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
0
votes
0answers
41 views

Calibrating LASSO prior (how to select the scale hyperparameter)?

I want to use a LASSO prior (Laplace prior) for a location parameter $\mu$ $$\pi(\mu \mid s) = \dfrac{1}{2s}\exp\left(-\frac{\vert \mu \vert}{s}\right).$$ However, I do not know to calibrate this ...
1
vote
1answer
61 views

James-Stein regularizing covariance like a mean

In James-Stein's estimator we have a $p$-dimensional random vector $X\sim N_{p}(\mu ,I)$ where $\mu \neq 0$ and the goal is to estimate the mean vector using the single ($n=1$) data vector $X$. The ...
0
votes
1answer
19 views

Prove that the direction in Least Angle Regression makes equal angle with all predictors [closed]

Least Angle Regression iteralively adds predictors according to the procedure described here : Writing by hand first steps in Least Angle Regression (LARS) We note $A_{k}$ the active set of variables ...
0
votes
1answer
72 views

Writing by hand first steps in Least Angle Regression (LARS)

How do we write the first steps of Least Angle Regression ? What is the rationale behind this method ? What limitations of other methods is it overcoming ? Why is it called Least Angle Regression ?
1
vote
1answer
54 views

What are some good examples of working through a multilevel model by hand?

I've been learning about multilevel models lately, and I understand the concept of shrinkage and partial pooling (I think), but I'm still confused to some extent on how partial pooling actually ...
3
votes
0answers
175 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
29 views

How to give a represantation to veriables from each group using LASSO

I'm trying to apply LASSO regression on my data set in order to choose the best variables. However, my variables (44 to be accurate) come from 7 different groups, is there any option to give a "...
2
votes
1answer
58 views

Soft thresholding (Donoho and Johnstone)

Donoho and Johnstone (1994) poses the following equality: $$ E((\eta_t(X) - \mu)^2) = 1 - 2\Pr(|X|\lt t) + E(\min(X^2,t^2)) $$ where $\eta_t(X) = \operatorname{sign}(x)\max(|X|-t,0)$ and $X \sim N(\...
0
votes
0answers
28 views

Is there any way to adjust (shrink) the unusually large predicted values after running a fixed effect model

I am trying to run a 2SLS and therefore, I need to use the predicted values from the 1st stage and use it in the second stage. Code from my 1st stage - ...
0
votes
0answers
75 views

Shrinkage methods - are they any good for statistical inference or should they be used for prediction goals only?

I am working on my master thesis with a goal to find regressors which influence companies' decisions on how to pay for a target in acquisitions (cash, stock or a mix of both). I have 13 regressors to ...
0
votes
1answer
236 views

Why lasso yield a higher mse then ridge?

I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction ...
4
votes
0answers
260 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
3
votes
1answer
279 views

Performance of Ridge and Lasso Regression depend on set.seed?

I try to do a ridge and lasso regression for out of sample predictions. The optimal lambda is chosed via cross validation. I run my results for different seeds in R. And depending on the seed i get a ...
1
vote
0answers
51 views

Ridge and Lasso Regression: Should I drop one reference category like in OLS? [duplicate]

I do a ridge and lasso regression with a data set that have categorial variables. Should i drop 1 reference category like in OLS or is it okay to run the regression with als categories as dummy ...
0
votes
0answers
47 views

Strange MSE as result in ridge and lasso regression

I did a lasso and ridge regression. In my data set i had p > n ( more variables then observations) . At the beginning of my analysis i had only 13 explenatory variables where some of the variables ...
6
votes
1answer
172 views

James-Stein Estimator with unequal variances (Ch. 2)

After studying James-Stein estimators for a few weeks and looking at many different sources I am stuck at trying to understand how Efron and Morris calculated the Toxoplasmosis example in their 1975 ...
3
votes
3answers
217 views

Why shrinkage estimators?

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction....
2
votes
0answers
124 views

LASSO: Why we have to center the dependent variable

I'm trying to get into the topic of shrinkage models. I don't unterstand why it is important to center y. The consequence is, that we can ommit the intercept. But why I should care if the model ...
0
votes
0answers
144 views

LASSO: Deriving intercept is equal to mean of y

I'm new in the world of mashine learning. My first project is trying to understand the mechanism of ridge/lasso regressions. Im stucked at the point how to derive that the intercept is equal to the ...
1
vote
0answers
22 views

Shrinking the mean: distance loss function and risk function

I'm reading some slides about the shrinkage of the mean and I cannot understand some results. Assume an n-dimensional vector $\mathbf{x} \sim N(\mu, I_n)$. We are interested in obtaining an estimate ...
1
vote
0answers
102 views

Elastic net consistency for time series case

I am looking for a paper that proves elastic net consistency (in estimation and model selection) for time series setting (non i.i.d. errors). I have found papers for LASSO and adaptive LASSO but after ...
1
vote
1answer
90 views

Marginal prior derivation in hierarchical Bayesian model

I am working on a model that is closely related to the normal gamma shrinkage prior setup discussed in Griffin & Brown (2010). Suppose we want to draw $P$ parameters $\beta_p$ with $p=1,...,P$. ...
5
votes
0answers
52 views

Do shrinkage estimators solve the Neyman-Scott paradox?

I read the following SE question: What problem do shrinkage methods solve? And I wondered if shrinkage estimators provide a consistent estimator of the sample variance in a "mixed-effects" model using ...
1
vote
0answers
347 views

Ridge regression / regularization approach to hierarchical model

Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...
2
votes
1answer
2k views

What Are Shrinking Heuristics

I have been working on a project with LibSVM and have noticed there is an option to train the SVM model with "shrinking heuristics" which are used to speed up the classifier training. After doing ...
1
vote
1answer
59 views

Unknown variance in a normal means model

I have data that I would like to model as Normal Means, i.e. with $X_i \sim \mathcal N(\theta_i,\sigma^2)$ for $i = 1,\ldots,n$, and I want an estimator of $\theta = (\theta_1,\ldots,\theta_i)$ that ...
47
votes
6answers
3k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
25
votes
1answer
1k views

Why does glmnet use “naive” elastic net from the Zou & Hastie original paper?

The original elastic net paper Zou & Hastie (2005) Regularization and variable selection via the elastic net introduced elastic net loss function for linear regression (here I assume all ...
8
votes
2answers
252 views

Feature selection on a Bayesian hierarchical generalized linear model

I am looking to estimate a hierarchical GLM but with feature selection to determine which covariates are relevant at the population level to include. Suppose I have $G$ groups with $N$ observations ...
1
vote
0answers
168 views

Overall shrinkage by bootstrap for multinomial regression

I am looking for a shrinkage technique which supplies an overall shrinkage factor for multinomial regression. I am building a risk prediction model in a medical setting for a 4-level outcome. I do ...
1
vote
1answer
168 views

Deriving posterior means for regression with horseshoe prior

I'm reading the Horseshoe prior regression paper which formulates regression like so: $(y|\beta) \sim N(\beta, \sigma^2I)$ $(\beta_i|\lambda_i,\tau) \sim N(0, \lambda_i^2 \tau^2)$ $\lambda_i \sim C^...
1
vote
0answers
72 views

Choice of ridge parameter in constrained estimation

Apologies for a possible basic question. Suppose I have a model $$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$ and, say, matrix $X'X$ is nearly singular and I have a ...
2
votes
1answer
44 views

Is there a generalization of the “within” transformation to partial pooling?

The random effects estimator for panel data can be cast as a penalized verion of the "fixed-effects" estimator from econometrics. In both cases, the model is $$ y_{it} = \mathbf{D}_{i}\alpha_i + \...
1
vote
0answers
445 views

Multi-collinearity and autocorrelation issues in applied regression problems

I have a problem regarding my final year research project which is developing a monetary model to predict the currency rate EURO/LKR using regression analysis. I used the multiple linear Regression ...
18
votes
1answer
4k views

Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
0
votes
0answers
75 views

Problem with optimum value for shrinkage parameter in glmnet in R

I have a query about the cv.glmnet() function in R which is supposed to find the "optimum" value of the parameter lambda for ridge regression. In the example code below, if you experiment a bit with ...
2
votes
2answers
78 views

Total Cost Shrinkage

I have a question regarding Shrinkage Methods. I am currently writing a term paper about ridge regression and lasso and before explaining the two methods, I want to give some theory on why shrinking ...
2
votes
1answer
68 views

James-Stein Estimator with unequal numbers in groups

In the book Computer Age Statistical Inference the James-Stein estimator is introduced. Brad Efron runs through an example where batting averages are estimated from each players 90 at-bats. $$p_i\sim ...
8
votes
3answers
1k views

In Ridge regression and LASSO, why smaller $\beta$ would be better?

Can anyone provide an intuitive view on why it is better to have smaller beta? For LASSO I can understand that, there is a feature selection component here. Less features make the model simpler and ...
7
votes
1answer
1k views

L2-regularization vs random effects shrinkage

A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate. ...
0
votes
1answer
88 views

Optimal Shrinkage with g-prior: estimate alpha to minimize MSE

In one of the slides from my class related to bayesian linear regression, I have the following scenario. Under g-prior, the shrinkage estimator induced by the prior is $$\hat{\beta_{\alpha}} = \alpha\...
3
votes
1answer
286 views

Why does my classical logistic regression model perform better than its elastic net counterpart?

I have about 200 observations and 33 predictors. Due to sample size limitation, I used an elastic net logistic regression model. I have really high specificity ~0.9 but really low sensitivity < 0....
11
votes
2answers
220 views

If shrinkage is applied in a clever way, does it always work better for more efficient estimators?

Suppose I have two estimators $\widehat{\beta}_1$ and $\widehat{\beta}_2$ that are consistent estimators of the same parameter $\beta_0$ and such that $$\sqrt{n}(\widehat{\beta}_1 -\beta_0) \stackrel{...
4
votes
1answer
172 views

What advantages does classical regression have over shrinkage methods?

Shrinkage approach like elastic net can produce predictive, sparse models when p >> n and when there's correlation between the predictors. In Zou and Hastie's paper on the elastic net, the simulation ...
0
votes
1answer
78 views

Is the shrinkage techique all about finite samples?

With the right choice of the penalty, the ridge estimator in the linear regression model $$y=X \beta +\varepsilon, \quad E[X \varepsilon]=0, \quad \beta \in \mathbb{R}^k$$ will have a smaller MSE ...
2
votes
1answer
396 views

Are ridge and LASSO shrinkage parameters functions of the number of features?

I was wondering if Ridge and LASSO shrinkage parameters (I am referring to the lambda in the canonical loss function) are functions of the number of features if one is targeting a "constant" shrinkage ...
6
votes
2answers
1k views

Using shrinkage when estimating covariance matrix before doing PCA

Although it is often calculated differently, my intuitive understanding of PCA arises from its definition as the eigendecomposition of the sample covariance matrix. I have recently become aware of ...
0
votes
1answer
69 views

Shrinkage in hierarhical models based not on observations

When we have a hierarchical model, such as: $$log(y_{i,t})=\beta_0 + \beta_i*log(x_{i,t})+\epsilon_{i,t}$$ Where $\beta_i$ ~ $N(B,\Sigma)$, and the sampling model is normal (normal disturbances.) ...
23
votes
3answers
6k views

LASSO with interaction terms - is it okay if main effects are shrunk to zero?

LASSO regression shrinks coefficients towards zero, thus providing effectively model selection. I believe that in my data there are meaningful interactions between nominal and continuous covariates. ...