Linked Questions

0
votes
1answer
11k views

What is penalized logistic regression [duplicate]

I need to do a logistic regression that will likely have a lot of zeros. Can someone explain penalized logistic regression to me like I'm dumb?
7
votes
3answers
2k views

Why does regularization of coefficient magnitude improve the generalization of linear regression? [duplicate]

What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that In many cases, tikhonov matrix is chosen as ...
2
votes
0answers
242 views

Why were 'Regularization' methods (Lasso, Ridge, Elastic Net) created in the first place? [duplicate]

What problem do regularization methods solve? I thought it was feature selection and to prevent overfitting. However, I was informed that the reason Ridge, Lasso, and Elastic Net were created in the ...
1
vote
0answers
217 views

Linear Regression — Regularization, shrinkage [duplicate]

Regularization reduces the magnitudes of the regression coefficients. I read that this helps reduce the variance of the model. Why exactly do smaller values of the coefficients lead to a lower ...
0
votes
1answer
50 views

Why is L2 regression good for handling multicollinearity? [duplicate]

Looking for an intuitive explanation, thanks.
1
vote
0answers
30 views

LASSO method. Intuitively how does it select variables? [duplicate]

Intuitively how does the LASSO method select its variables? Is it based on standard econometrics?
25
votes
5answers
4k views

How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?

Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$? I understand that from dimensionality-reduction/...
46
votes
2answers
4k views

Intuition behind why Stein's paradox only applies in dimensions $\ge 3$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
36
votes
2answers
9k views

When is a biased estimator preferable to unbiased one?

It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?
30
votes
2answers
4k views

Theory behind partial least squares regression

Can anyone recommend a good exposition of the theory behind partial least squares regression (available online) for someone who understands SVD and PCA? I have looked at many sources online and have ...
38
votes
3answers
5k views

Empirical justification for the one standard error rule when using cross-validation

Are there any empirical studies justifying the use of the one standard error rule in favour of parsimony? Obviously it depends on the data-generation process of the data, but anything which analyses a ...
13
votes
2answers
2k views

Why is best subset selection not favored in comparison to lasso?

I'm reading about best subset selection in the Elements of statistical learning book. If I have 3 predictors $x_1,x_2,x_3$, I create $2^3=8$ subsets: Subset with no predictors subset with predictor $...
12
votes
1answer
1k views

Why does Daniel Wilks (2011) say that principal component regression “will be biased”?

In Statistical Methods in the Atmospheric Sciences, Daniel Wilks notes that multiple linear regression can lead to problems if there are very strong intercorrelations among the predictors (3rd edition,...
15
votes
2answers
342 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
5
votes
1answer
483 views

Where is there bias-variance trade-off, and why?

In Wikipedia, the "Bias–variance tradeoff" is mentioned in the context of prediction models where one can control the complexity of the model with some tuning parameters, and the more complex the ...

15 30 50 per page