Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [shrinkage]

Shrinkage in statistics is a form of regularization consisting in changing parameter estimates to be "smaller" (closer to zero), or, more generally, "closer to each other".

Filter by
Sorted by
Tagged with
1
vote
0answers
27 views

Lasso: more penalization with more data?

I am currently doing a backtest of a financial data set with an expanding window. For this, I estimate a Lasso model each month. Hence, each month that I estimate the model, I will have more data. ...
0
votes
0answers
41 views

Deriving posterior mean with horseshoe prior

I want to decompose a matrix $S \in \mathbb{R}^{D \times D}$ as below $$S=vv^T $$ where $v_i\mid\lambda_i,\tau_i \sim N(0,\lambda^2_i\rho^2_i)$, $\lambda_i \sim Cauchy^+(0,1)$ i.e $v$ has horseshoe ...
0
votes
0answers
25 views

How to Create a Toy Example of the Curds and Whey Algorithm?

Why is my simulated example failing? I am trying to create a toy example of the Curds and Whey method for multivariate linear regression in python (An example in R would be very helpful as well). I ...
0
votes
0answers
15 views

How to get change of basis matrix for Canonical Correlation Analysis?

A bit of background: I am trying to create toy example of the Curds and Whey regression shrinkage algorithm in python. In a standard multivariate regression this algorithm uses canonical correlation ...
1
vote
0answers
36 views

Long-run covariance matrix estimators with Ledoit-Wolf (2004) shrinkage; what methods exist?

Ledoit and Wolf ("A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices", 2004) proposed an estimator for the covariance matrix of a data set, $S^* = p I_d + (1 - p) \hat{S}$ with $p \...
0
votes
0answers
34 views

Shrinkage Parameter for LASSO (glmnet package in R)

I am using glmnet package in R-software to build a Binary Logistic Regression with the LASSO. I have used the following link as ...
2
votes
0answers
62 views

James Stein Estimator for more than one Sample

I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
15
votes
2answers
372 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
0
votes
0answers
43 views

Calibrating LASSO prior (how to select the scale hyperparameter)?

I want to use a LASSO prior (Laplace prior) for a location parameter $\mu$ $$\pi(\mu \mid s) = \dfrac{1}{2s}\exp\left(-\frac{\vert \mu \vert}{s}\right).$$ However, I do not know to calibrate this ...
2
votes
1answer
112 views

James-Stein regularizing covariance like a mean

In James-Stein's estimator we have a $p$-dimensional random vector $X\sim N_{p}(\mu ,I)$ where $\mu \neq 0$ and the goal is to estimate the mean vector using the single ($n=1$) data vector $X$. The ...
2
votes
1answer
14 views

Inflation of effect size estimation among significant results?

Correction for multiple testing is often performed on the alpha value. However, multiple testing is, I think, likely to inflate also the estimated effect size of the most significant effects found. ...
0
votes
1answer
27 views

Prove that the direction in Least Angle Regression makes equal angle with all predictors [closed]

Least Angle Regression iteralively adds predictors according to the procedure described here : Writing by hand first steps in Least Angle Regression (LARS) We note $A_{k}$ the active set of variables ...
0
votes
1answer
129 views

Writing by hand first steps in Least Angle Regression (LARS)

How do we write the first steps of Least Angle Regression ? What is the rationale behind this method ? What limitations of other methods is it overcoming ? Why is it called Least Angle Regression ?
1
vote
1answer
60 views

What are some good examples of working through a multilevel model by hand?

I've been learning about multilevel models lately, and I understand the concept of shrinkage and partial pooling (I think), but I'm still confused to some extent on how partial pooling actually ...
3
votes
0answers
242 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0
votes
0answers
32 views

How to give a represantation to veriables from each group using LASSO

I'm trying to apply LASSO regression on my data set in order to choose the best variables. However, my variables (44 to be accurate) come from 7 different groups, is there any option to give a "...
2
votes
1answer
68 views

Soft thresholding (Donoho and Johnstone)

Donoho and Johnstone (1994) poses the following equality: $$ E((\eta_t(X) - \mu)^2) = 1 - 2\Pr(|X|\lt t) + E(\min(X^2,t^2)) $$ where $\eta_t(X) = \operatorname{sign}(x)\max(|X|-t,0)$ and $X \sim N(\...
0
votes
0answers
30 views

Is there any way to adjust (shrink) the unusually large predicted values after running a fixed effect model

I am trying to run a 2SLS and therefore, I need to use the predicted values from the 1st stage and use it in the second stage. Code from my 1st stage - ...
0
votes
0answers
86 views

Shrinkage methods - are they any good for statistical inference or should they be used for prediction goals only?

I am working on my master thesis with a goal to find regressors which influence companies' decisions on how to pay for a target in acquisitions (cash, stock or a mix of both). I have 13 regressors to ...
0
votes
1answer
290 views

Why lasso yield a higher mse then ridge?

I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction ...
4
votes
0answers
344 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
3
votes
1answer
361 views

Performance of Ridge and Lasso Regression depend on set.seed?

I try to do a ridge and lasso regression for out of sample predictions. The optimal lambda is chosed via cross validation. I run my results for different seeds in R. And depending on the seed i get a ...
1
vote
0answers
51 views

Ridge and Lasso Regression: Should I drop one reference category like in OLS? [duplicate]

I do a ridge and lasso regression with a data set that have categorial variables. Should i drop 1 reference category like in OLS or is it okay to run the regression with als categories as dummy ...
0
votes
0answers
57 views

Strange MSE as result in ridge and lasso regression

I did a lasso and ridge regression. In my data set i had p > n ( more variables then observations) . At the beginning of my analysis i had only 13 explenatory variables where some of the variables ...
6
votes
1answer
193 views

James-Stein Estimator with unequal variances (Ch. 2)

After studying James-Stein estimators for a few weeks and looking at many different sources I am stuck at trying to understand how Efron and Morris calculated the Toxoplasmosis example in their 1975 ...
3
votes
3answers
262 views

Why shrinkage estimators?

Iam trying to understand the usage of lasso and ridge regression. The advantage of both methods is that we get a lower variance in comparisson to the ols estimation and thus we get a better prediction....
2
votes
0answers
149 views

LASSO: Why we have to center the dependent variable

I'm trying to get into the topic of shrinkage models. I don't unterstand why it is important to center y. The consequence is, that we can ommit the intercept. But why I should care if the model ...
0
votes
0answers
186 views

LASSO: Deriving intercept is equal to mean of y

I'm new in the world of mashine learning. My first project is trying to understand the mechanism of ridge/lasso regressions. Im stucked at the point how to derive that the intercept is equal to the ...
1
vote
0answers
25 views

Shrinking the mean: distance loss function and risk function

I'm reading some slides about the shrinkage of the mean and I cannot understand some results. Assume an n-dimensional vector $\mathbf{x} \sim N(\mu, I_n)$. We are interested in obtaining an estimate ...
1
vote
0answers
133 views

Elastic net consistency for time series case

I am looking for a paper that proves elastic net consistency (in estimation and model selection) for time series setting (non i.i.d. errors). I have found papers for LASSO and adaptive LASSO but after ...
1
vote
1answer
97 views

Marginal prior derivation in hierarchical Bayesian model

I am working on a model that is closely related to the normal gamma shrinkage prior setup discussed in Griffin & Brown (2010). Suppose we want to draw $P$ parameters $\beta_p$ with $p=1,...,P$. ...
5
votes
0answers
55 views

Do shrinkage estimators solve the Neyman-Scott paradox?

I read the following SE question: What problem do shrinkage methods solve? And I wondered if shrinkage estimators provide a consistent estimator of the sample variance in a "mixed-effects" model using ...
1
vote
0answers
408 views

Ridge regression / regularization approach to hierarchical model

Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...
2
votes
1answer
3k views

What Are Shrinking Heuristics

I have been working on a project with LibSVM and have noticed there is an option to train the SVM model with "shrinking heuristics" which are used to speed up the classifier training. After doing ...
1
vote
1answer
60 views

Unknown variance in a normal means model

I have data that I would like to model as Normal Means, i.e. with $X_i \sim \mathcal N(\theta_i,\sigma^2)$ for $i = 1,\ldots,n$, and I want an estimator of $\theta = (\theta_1,\ldots,\theta_i)$ that ...
50
votes
6answers
4k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
27
votes
1answer
1k views

Why does glmnet use “naive” elastic net from the Zou & Hastie original paper?

The original elastic net paper Zou & Hastie (2005) Regularization and variable selection via the elastic net introduced elastic net loss function for linear regression (here I assume all ...
8
votes
2answers
287 views

Feature selection on a Bayesian hierarchical generalized linear model

I am looking to estimate a hierarchical GLM but with feature selection to determine which covariates are relevant at the population level to include. Suppose I have $G$ groups with $N$ observations ...
1
vote
0answers
187 views

Overall shrinkage by bootstrap for multinomial regression

I am looking for a shrinkage technique which supplies an overall shrinkage factor for multinomial regression. I am building a risk prediction model in a medical setting for a 4-level outcome. I do ...
1
vote
1answer
181 views

Deriving posterior means for regression with horseshoe prior

I'm reading the Horseshoe prior regression paper which formulates regression like so: $(y|\beta) \sim N(\beta, \sigma^2I)$ $(\beta_i|\lambda_i,\tau) \sim N(0, \lambda_i^2 \tau^2)$ $\lambda_i \sim C^...
1
vote
0answers
74 views

Choice of ridge parameter in constrained estimation

Apologies for a possible basic question. Suppose I have a model $$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$ and, say, matrix $X'X$ is nearly singular and I have a ...
2
votes
1answer
44 views

Is there a generalization of the “within” transformation to partial pooling?

The random effects estimator for panel data can be cast as a penalized verion of the "fixed-effects" estimator from econometrics. In both cases, the model is $$ y_{it} = \mathbf{D}_{i}\alpha_i + \...
1
vote
0answers
472 views

Multi-collinearity and autocorrelation issues in applied regression problems

I have a problem regarding my final year research project which is developing a monetary model to predict the currency rate EURO/LKR using regression analysis. I used the multiple linear Regression ...
18
votes
1answer
5k views

Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
0
votes
0answers
92 views

Problem with optimum value for shrinkage parameter in glmnet in R

I have a query about the cv.glmnet() function in R which is supposed to find the "optimum" value of the parameter lambda for ridge regression. In the example code below, if you experiment a bit with ...
2
votes
2answers
83 views

Total Cost Shrinkage

I have a question regarding Shrinkage Methods. I am currently writing a term paper about ridge regression and lasso and before explaining the two methods, I want to give some theory on why shrinking ...
2
votes
1answer
83 views

James-Stein Estimator with unequal numbers in groups

In the book Computer Age Statistical Inference the James-Stein estimator is introduced. Brad Efron runs through an example where batting averages are estimated from each players 90 at-bats. $$p_i\sim ...
8
votes
3answers
1k views

In Ridge regression and LASSO, why smaller $\beta$ would be better?

Can anyone provide an intuitive view on why it is better to have smaller beta? For LASSO I can understand that, there is a feature selection component here. Less features make the model simpler and ...
8
votes
1answer
1k views

L2-regularization vs random effects shrinkage

A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate. ...
0
votes
1answer
90 views

Optimal Shrinkage with g-prior: estimate alpha to minimize MSE

In one of the slides from my class related to bayesian linear regression, I have the following scenario. Under g-prior, the shrinkage estimator induced by the prior is $$\hat{\beta_{\alpha}} = \alpha\...