Linked Questions

44
votes
6answers
19k views

Intuitive explanation of the bias-variance tradeoff?

I am looking for an intuitive explanation of the bias-variance tradeoff, both in general and specifically in the context of linear regression.
21
votes
3answers
10k views

Why use Lasso estimates over OLS estimates on the Lasso-identified subset of variables?

For Lasso regression $$L(\beta)=(X\beta-y)'(X\beta-y)+\lambda\|\beta\|_1,$$ suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\...
18
votes
1answer
4k views

Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
12
votes
3answers
4k views

How can you handle unstable $\beta$ estimates in linear regression with high multi-collinearity without throwing out variables?

Beta stability in linear regression with high multi-collinearity? Let's say in a linear regression, the variables $x_1$ and $x_2$ has high multi-collinearity (correlation is around 0.9). We are ...
7
votes
4answers
2k views

Why must one trade off between bias and variance?

Apparently, a learning algorithm must make a trade off between bias and variance when producing a hypothesis. Bias means systematic deviation from data. Variance refers to the error due to ...
13
votes
4answers
3k views

Optimal penalty selection for lasso

Are there any analytical results or experimental papers regarding the optimal choice of the coefficient of the $\ell_1$ penalty term. By optimal, I mean a parameter that maximizes the probability of ...
4
votes
2answers
2k views

Error increase on L2 regularization in an NN

When introducing L2 regularization on my neural network, there is a point during training where the error starts to increase after having reached a value very close to 0. This is due to the fact that ...
7
votes
1answer
4k views

Is Bayesian Ridge Regression another name of Bayesian Linear Regression?

I searched about Bayesian Ridge Regression on Internet but most of the result i became is about Bayesian Linear Regression. I wonder if it's both the same things because the formula look quite similar
8
votes
1answer
9k views

Omitted variable bias in linear regression

I have a philosophical question regarding omitted variable bias. We have the typical regression model (population model) $$ Y= \beta_0 + \beta_1X_1 + ... + \beta_nX_n + \upsilon, $$ where the ...
1
vote
1answer
6k views

How to interprete lasso from lars correctly?

I tried the lars package with R and got the following result. ...
2
votes
1answer
6k views

Cohen's d and multiple comparisons for 2/3-way ANOVA

I am conducting three-way ANOVA (A*B*C) with 2 levels each. 1) I found A*B interaction. 2) I moved to 2-way ANOVA (A*B) and found interaction again. I reported Eta-squared and equivalent Cohen's d. 3) ...
6
votes
1answer
423 views

When will a less true model predict better than a truer model?

In "To Explain or to Predict?", Pr. Galit Shmueli said that sometimes a less true model can predict better than a truer model. Why is it so? When will it happen? How does it happen? Is explanation a ...
8
votes
2answers
469 views

Do stepwise regression techniques increase a model's predictive power?

I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the ...
18
votes
2answers
314 views

Can regularization be helpful if we are interested only in modeling, not in forecasting?

Can regularization be helpful if we are interested only in estimating (and interpreting) the model parameters, not in forecasting or prediction? I see how regularization/cross-validation is extremely ...
3
votes
1answer
2k views

logistic regression in r with many predictors

I have been running logistic regression in R, and have been having an issue where as I include more predictors the z-scores and respective p-values approach 0 and 1 respectively. For example if have ...

15 30 50 per page