Linked Questions
22 questions linked to/from What problem do shrinkage methods solve?
61
votes
7
answers
27k
views
Intuitive explanation of the bias-variance tradeoff?
I am looking for an intuitive explanation of the bias-variance tradeoff, both in general and specifically in the context of linear regression.
- 5,491
34
votes
3
answers
20k
views
Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables?
For Lasso regression $$L(\beta)=(X\beta-y)'(X\beta-y)+\lambda\|\beta\|_1,$$ suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\...
- 855
28
votes
3
answers
13k
views
Inference after using Lasso for variable selection
I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no ...
- 2,603
23
votes
1
answer
11k
views
Why is the James-Stein estimator called a "shrinkage" estimator?
I have been reading about the James-Stein estimator. It is defined, in this notes, as
$$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$
I have read the proof but I don't understand the ...
- 1,676
13
votes
3
answers
6k
views
How can you handle unstable $\beta$ estimates in linear regression with high multi-collinearity without throwing out variables?
Beta stability in linear regression with high multi-collinearity?
Let's say in a linear regression, the variables $x_1$ and $x_2$ has high multi-collinearity (correlation is around 0.9).
We are ...
- 2,335
14
votes
1
answer
18k
views
Is Bayesian Ridge Regression another name of Bayesian Linear Regression?
I searched about Bayesian Ridge Regression on Internet but most of the result I got is about Bayesian Linear Regression. I wonder if it's both the same things because the formula look quite similar
- 315
16
votes
4
answers
5k
views
Optimal penalty selection for lasso
Are there any analytical results or experimental papers regarding the optimal choice of the coefficient of the $\ell_1$ penalty term. By optimal, I mean a parameter that maximizes the probability of ...
- 5,490
8
votes
4
answers
3k
views
Why must one trade off between bias and variance?
Apparently, a learning algorithm must make a trade off between bias and variance when producing a hypothesis. Bias means systematic deviation from data. Variance refers to the error due to ...
- 943
4
votes
2
answers
4k
views
Error increase on L2 regularization in an NN
When introducing L2 regularization on my neural network, there is a point during training where the error starts to increase after having reached a value very close to 0. This is due to the fact that ...
- 43
12
votes
1
answer
11k
views
Omitted variable bias in linear regression
I have a philosophical question regarding omitted variable bias.
We have the typical regression model (population model)
$$
Y= \beta_0 + \beta_1X_1 + ... + \beta_nX_n + \upsilon,
$$
where the ...
- 123
10
votes
1
answer
697
views
When will a less true model predict better than a truer model?
In "To Explain or to Predict?", Pr. Galit Shmueli said that sometimes a less true model can predict better than a truer model. Why is it so? When will it happen? How does it happen? Is ...
- 1,566
4
votes
1
answer
4k
views
Which ML Algorithms are affected by dummy variable trap?
My understanding is that regression models are affected by the dummy variable trap. What about other machine learning algorithms e.g. linear svm, logistic regression?
Also, if an algorithm is not ...
2
votes
1
answer
9k
views
How to interprete lasso from lars correctly?
I tried the lars package with R and got the following result.
...
- 705
8
votes
2
answers
835
views
Do stepwise regression techniques increase a model's predictive power?
I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the ...
- 4,069
2
votes
1
answer
8k
views
Cohen's d and multiple comparisons for 2/3-way ANOVA
I am conducting three-way ANOVA (A*B*C) with 2 levels each.
1) I found A*B interaction.
2) I moved to 2-way ANOVA (A*B) and found interaction again. I reported Eta-squared and equivalent Cohen's d.
3) ...
- 109