Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [shrinkage]

Shrinkage in statistics is a form of regularization consisting in changing parameter estimates to be "smaller" (closer to zero), or, more generally, "closer to each other".

0
votes
0answers
34 views

Calibrating LASSO prior (how to select the scale hyperparameter)?

I want to use a LASSO prior (Laplace prior) for a location parameter $\mu$ $$\pi(\mu \mid s) = \dfrac{1}{2s}\exp\left(-\frac{\vert \mu \vert}{s}\right).$$ However, I do not know to calibrate this ...
1
vote
1answer
53 views

James-Stein regularizing covariance like a mean

In James-Stein's estimator we have a $p$-dimensional random vector $X\sim N_{p}(\mu ,I)$ where $\mu \neq 0$ and the goal is to estimate the mean vector using the single ($n=1$) data vector $X$. The ...
0
votes
1answer
16 views

Prove that the direction in Least Angle Regression makes equal angle with all predictors [closed]

Least Angle Regression iteralively adds predictors according to the procedure described here : Writing by hand first steps in Least Angle Regression (LARS) We note $A_{k}$ the active set of variables ...
0
votes
1answer
28 views

Writing by hand first steps in Least Angle Regression (LARS)

How do we write the first steps of Least Angle Regression ? What is the rationale behind this method ? What limitations of other methods is it overcoming ? Why is it called Least Angle Regression ?
1
vote
1answer
47 views

What are some good examples of working through a multilevel model by hand?

I've been learning about multilevel models lately, and I understand the concept of shrinkage and partial pooling (I think), but I'm still confused to some extent on how partial pooling actually ...
2
votes
0answers
115 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
24 views

How to give a represantation to veriables from each group using LASSO

I'm trying to apply LASSO regression on my data set in order to choose the best variables. However, my variables (44 to be accurate) come from 7 different groups, is there any option to give a "...
2
votes
1answer
53 views

Soft thresholding (Donoho and Johnstone)

Donoho and Johnstone (1994) poses the following equality: $$ E((\eta_t(X) - \mu)^2) = 1 - 2\Pr(|X|\lt t) + E(\min(X^2,t^2)) $$ where $\eta_t(X) = \operatorname{sign}(x)\max(|X|-t,0)$ and $X \sim N(\...
0
votes
0answers
27 views

Is there any way to adjust (shrink) the unusually large predicted values after running a fixed effect model

I am trying to run a 2SLS and therefore, I need to use the predicted values from the 1st stage and use it in the second stage. Code from my 1st stage - ...
0
votes
0answers
67 views

Shrinkage methods - are they any good for statistical inference or should they be used for prediction goals only?

I am working on my master thesis with a goal to find regressors which influence companies' decisions on how to pay for a target in acquisitions (cash, stock or a mix of both). I have 13 regressors to ...
0
votes
1answer
202 views

Why lasso yield a higher mse then ridge?

I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction ...
4
votes
0answers
223 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
0
votes
0answers
77 views

Why is train MSE bigger then test MSE?

I did a lasso and ridge regression. In both cases my train mse is bigger then my test mse and i ask myself why ?
0
votes
0answers
68 views

Bigger K for Cross validation ridge regression is better?

I do a ridge Regression and choose my Lambda via cross validation. When i set k = 10 in cross validation allways get smaller mse on test and training data set comparing to set k = 5. What is the ...
3
votes
1answer
239 views

Performance of Ridge and Lasso Regression depend on set.seed?

I try to do a ridge and lasso regression for out of sample predictions. The optimal lambda is chosed via cross validation. I run my results for different seeds in R. And depending on the seed i get a ...
1
vote
0answers
49 views

Ridge and Lasso Regression: Should I drop one reference category like in OLS? [duplicate]

I do a ridge and lasso regression with a data set that have categorial variables. Should i drop 1 reference category like in OLS or is it okay to run the regression with als categories as dummy ...
0
votes
0answers
44 views

Strange MSE as result in ridge and lasso regression

I did a lasso and ridge regression. In my data set i had p > n ( more variables then observations) . At the beginning of my analysis i had only 13 explenatory variables where some of the variables ...
0
votes
0answers
20 views

Implementation of shrinkage parameter

I am trying to (better) understand the shrinkage parameter of a regularized canonical correlation analysis, as implemented in RGCCA package. The function in the package is described as following: <...
6
votes
1answer
157 views

James-Stein Estimator with unequal variances (Ch. 2)

After studying James-Stein estimators for a few weeks and looking at many different sources I am stuck at trying to understand how Efron and Morris calculated the Toxoplasmosis example in their 1975 ...
0
votes
0answers
29 views

Why i bias is not so bad for out of sample predictions?

Lets say i want to do a out of sample prediction with lasso or ridge regression. Both models produce coefficents that are biased but have a smaller variance the the ols coeffients for example. Why ...
3
votes
3answers
188 views

Why shrinkage estimators?

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction....
2
votes
0answers
104 views

LASSO: Why we have to center the dependent variable

I'm trying to get into the topic of shrinkage models. I don't unterstand why it is important to center y. The consequence is, that we can ommit the intercept. But why I should care if the model ...
0
votes
0answers
124 views

LASSO: Deriving intercept is equal to mean of y

I'm new in the world of mashine learning. My first project is trying to understand the mechanism of ridge/lasso regressions. Im stucked at the point how to derive that the intercept is equal to the ...
1
vote
0answers
21 views

Shrinking the mean: distance loss function and risk function

I'm reading some slides about the shrinkage of the mean and I cannot understand some results. Assume an n-dimensional vector $\mathbf{x} \sim N(\mu, I_n)$. We are interested in obtaining an estimate ...
1
vote
0answers
84 views

Elastic net consistency for time series case

I am looking for a paper that proves elastic net consistency (in estimation and model selection) for time series setting (non i.i.d. errors). I have found papers for LASSO and adaptive LASSO but after ...
1
vote
1answer
83 views

Marginal prior derivation in hierarchical Bayesian model

I am working on a model that is closely related to the normal gamma shrinkage prior setup discussed in Griffin & Brown (2010). Suppose we want to draw $P$ parameters $\beta_p$ with $p=1,...,P$. ...
5
votes
0answers
50 views

Do shrinkage estimators solve the Neyman-Scott paradox?

I read the following SE question: What problem do shrinkage methods solve? And I wondered if shrinkage estimators provide a consistent estimator of the sample variance in a "mixed-effects" model using ...
1
vote
0answers
315 views

Ridge regression / regularization approach to hierarchical model

Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...
1
vote
1answer
2k views

What Are Shrinking Heuristics

I have been working on a project with LibSVM and have noticed there is an option to train the SVM model with "shrinking heuristics" which are used to speed up the classifier training. After doing ...
1
vote
1answer
58 views

Unknown variance in a normal means model

I have data that I would like to model as Normal Means, i.e. with $X_i \sim \mathcal N(\theta_i,\sigma^2)$ for $i = 1,\ldots,n$, and I want an estimator of $\theta = (\theta_1,\ldots,\theta_i)$ that ...
48
votes
6answers
3k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
26
votes
1answer
1k views

Why does glmnet use “naive” elastic net from the Zou & Hastie original paper?

The original elastic net paper Zou & Hastie (2005) Regularization and variable selection via the elastic net introduced elastic net loss function for linear regression (here I assume all ...
8
votes
2answers
233 views

Feature selection on a Bayesian hierarchical generalized linear model

I am looking to estimate a hierarchical GLM but with feature selection to determine which covariates are relevant at the population level to include. Suppose I have $G$ groups with $N$ observations ...
1
vote
0answers
154 views

Overall shrinkage by bootstrap for multinomial regression

I am looking for a shrinkage technique which supplies an overall shrinkage factor for multinomial regression. I am building a risk prediction model in a medical setting for a 4-level outcome. I do ...
1
vote
1answer
155 views

Deriving posterior means for regression with horseshoe prior

I'm reading the Horseshoe prior regression paper which formulates regression like so: $(y|\beta) \sim N(\beta, \sigma^2I)$ $(\beta_i|\lambda_i,\tau) \sim N(0, \lambda_i^2 \tau^2)$ $\lambda_i \sim C^...
1
vote
0answers
72 views

Choice of ridge parameter in constrained estimation

Apologies for a possible basic question. Suppose I have a model $$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$ and, say, matrix $X'X$ is nearly singular and I have a ...
2
votes
1answer
44 views

Is there a generalization of the “within” transformation to partial pooling?

The random effects estimator for panel data can be cast as a penalized verion of the "fixed-effects" estimator from econometrics. In both cases, the model is $$ y_{it} = \mathbf{D}_{i}\alpha_i + \...
1
vote
0answers
415 views

Multi-collinearity and autocorrelation issues in applied regression problems

I have a problem regarding my final year research project which is developing a monetary model to predict the currency rate EURO/LKR using regression analysis. I used the multiple linear Regression ...
18
votes
1answer
4k views

Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
0
votes
0answers
66 views

Problem with optimum value for shrinkage parameter in glmnet in R

I have a query about the cv.glmnet() function in R which is supposed to find the "optimum" value of the parameter lambda for ridge regression. In the example code below, if you experiment a bit with ...
2
votes
2answers
78 views

Total Cost Shrinkage

I have a question regarding Shrinkage Methods. I am currently writing a term paper about ridge regression and lasso and before explaining the two methods, I want to give some theory on why shrinking ...
2
votes
1answer
62 views

James-Stein Estimator with unequal numbers in groups

In the book Computer Age Statistical Inference the James-Stein estimator is introduced. Brad Efron runs through an example where batting averages are estimated from each players 90 at-bats. $$p_i\sim ...
8
votes
3answers
1k views

In Ridge regression and LASSO, why smaller $\beta$ would be better?

Can anyone provide an intuitive view on why it is better to have smaller beta? For LASSO I can understand that, there is a feature selection component here. Less features make the model simpler and ...
7
votes
1answer
1k views

L2-regularization vs random effects shrinkage

A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate. ...
0
votes
1answer
86 views

Optimal Shrinkage with g-prior: estimate alpha to minimize MSE

In one of the slides from my class related to bayesian linear regression, I have the following scenario. Under g-prior, the shrinkage estimator induced by the prior is $$\hat{\beta_{\alpha}} = \alpha\...
3
votes
1answer
270 views

Why does my classical logistic regression model perform better than its elastic net counterpart?

I have about 200 observations and 33 predictors. Due to sample size limitation, I used an elastic net logistic regression model. I have really high specificity ~0.9 but really low sensitivity < 0....
11
votes
2answers
219 views

If shrinkage is applied in a clever way, does it always work better for more efficient estimators?

Suppose I have two estimators $\widehat{\beta}_1$ and $\widehat{\beta}_2$ that are consistent estimators of the same parameter $\beta_0$ and such that $$\sqrt{n}(\widehat{\beta}_1 -\beta_0) \stackrel{...
4
votes
1answer
159 views

What advantages does classical regression have over shrinkage methods?

Shrinkage approach like elastic net can produce predictive, sparse models when p >> n and when there's correlation between the predictors. In Zou and Hastie's paper on the elastic net, the simulation ...
0
votes
1answer
73 views

Is the shrinkage techique all about finite samples?

With the right choice of the penalty, the ridge estimator in the linear regression model $$y=X \beta +\varepsilon, \quad E[X \varepsilon]=0, \quad \beta \in \mathbb{R}^k$$ will have a smaller MSE ...
3
votes
1answer
395 views

Are ridge and LASSO shrinkage parameters functions of the number of features?

I was wondering if Ridge and LASSO shrinkage parameters (I am referring to the lambda in the canonical loss function) are functions of the number of features if one is targeting a "constant" shrinkage ...