Linked Questions

61 votes
7 answers

Intuitive explanation of the bias-variance tradeoff?

I am looking for an intuitive explanation of the bias-variance tradeoff, both in general and specifically in the context of linear regression.
NPE's user avatar
  • 5,491
34 votes
3 answers

Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables?

For Lasso regression $$L(\beta)=(X\beta-y)'(X\beta-y)+\lambda\|\beta\|_1,$$ suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\...
yliueagle's user avatar
  • 855
28 votes
3 answers

Inference after using Lasso for variable selection

I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no ...
Eli's user avatar
  • 2,603
23 votes
1 answer

Why is the James-Stein estimator called a "shrinkage" estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
3x89g2's user avatar
  • 1,676
13 votes
3 answers

How can you handle unstable $\beta$ estimates in linear regression with high multi-collinearity without throwing out variables?

Beta stability in linear regression with high multi-collinearity? Let's say in a linear regression, the variables $x_1$ and $x_2$ has high multi-collinearity (correlation is around 0.9). We are ...
Luna's user avatar
  • 2,335
14 votes
1 answer

Is Bayesian Ridge Regression another name of Bayesian Linear Regression?

I searched about Bayesian Ridge Regression on Internet but most of the result I got is about Bayesian Linear Regression. I wonder if it's both the same things because the formula look quite similar
Thien's user avatar
  • 315
16 votes
4 answers

Optimal penalty selection for lasso

Are there any analytical results or experimental papers regarding the optimal choice of the coefficient of the $\ell_1$ penalty term. By optimal, I mean a parameter that maximizes the probability of ...
gappy's user avatar
  • 5,490
8 votes
4 answers

Why must one trade off between bias and variance?

Apparently, a learning algorithm must make a trade off between bias and variance when producing a hypothesis. Bias means systematic deviation from data. Variance refers to the error due to ...
David Faux's user avatar
4 votes
2 answers

Error increase on L2 regularization in an NN

When introducing L2 regularization on my neural network, there is a point during training where the error starts to increase after having reached a value very close to 0. This is due to the fact that ...
Prasqui's user avatar
  • 43
12 votes
1 answer

Omitted variable bias in linear regression

I have a philosophical question regarding omitted variable bias. We have the typical regression model (population model) $$ Y= \beta_0 + \beta_1X_1 + ... + \beta_nX_n + \upsilon, $$ where the ...
Josu Momediano's user avatar
10 votes
1 answer

When will a less true model predict better than a truer model?

In "To Explain or to Predict?", Pr. Galit Shmueli said that sometimes a less true model can predict better than a truer model. Why is it so? When will it happen? How does it happen? Is ...
KuJ's user avatar
  • 1,566
4 votes
1 answer

Which ML Algorithms are affected by dummy variable trap?

My understanding is that regression models are affected by the dummy variable trap. What about other machine learning algorithms e.g. linear svm, logistic regression? Also, if an algorithm is not ...
Stamatis Tiniakos's user avatar
2 votes
1 answer

How to interprete lasso from lars correctly?

I tried the lars package with R and got the following result. ...
Carol.Kar's user avatar
  • 705
8 votes
2 answers

Do stepwise regression techniques increase a model's predictive power?

I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the ...
Underminer's user avatar
  • 4,069
2 votes
1 answer

Cohen's d and multiple comparisons for 2/3-way ANOVA

I am conducting three-way ANOVA (A*B*C) with 2 levels each. 1) I found A*B interaction. 2) I moved to 2-way ANOVA (A*B) and found interaction again. I reported Eta-squared and equivalent Cohen's d. 3) ...
JonBonJovi's user avatar

15 30 50 per page