Linked Questions

43
votes
6answers
3k views

What are the main theorems in Machine (Deep) Learning?

Al Rahimi has recently given a very provocative talk in NIPS 2017 comparing current Machine Learning to Alchemy. One of his claims is that we need to get back to theoretical developments, to have ...
47
votes
6answers
3k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
9
votes
1answer
3k views

Are estimates of regression coefficients uncorrelated?

Consider a simple regression (normality not assumed): $$Y_i = a + b X_i + e_i,$$ where $e_i$ is with mean $0$ and standard deviation $\sigma$. Are the Least Square Estimates of $a$ and $b$ ...
16
votes
1answer
3k views

Under exactly what conditions is ridge regression able to provide an improvement over ordinary least squares regression?

Ridge regression estimates parameters $\boldsymbol \beta$ in a linear model $\mathbf y = \mathbf X \boldsymbol \beta$ by $$\hat{\boldsymbol \beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf ...
11
votes
1answer
2k views

James-Stein Estimator with unequal variances

Every statement I find of the James-Stein estimator assumes that the random variables being estimated have the same (and unit) variance. But all of these examples also mention that the JS estimator ...
12
votes
1answer
2k views

Is there a connection between empirical Bayes and random effects?

I recently happened to read about empirical Bayes (Casella, 1985, An introduction to empirical Bayes data analysis) and it looked a lot like random effects model; in that both have estimates shrunken ...
7
votes
1answer
2k views

Estimates of random effects in binomial model (lme4)

I'm simulating Bernoulli trials with a random $\text{logit}\, \theta \sim {\cal N}(\text{logit}\, \theta_0, 1^2)$ between groups and then I fit the corresponding model with the ...
4
votes
1answer
261 views

What is the rationale behind LARS-OLS hybrid, i.e. using OLS estimate on the variables chosen by LARS?

I need some help to understand the relationship between the ranking of the variables from the LARS algorithm and the use of OLS to estimate the final model chosen by the LARS. I understand that the ...
1
vote
0answers
349 views

Ridge regression / regularization approach to hierarchical model

Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...
1
vote
0answers
139 views

Difference between random effect and fixed effect with regularization/prior

Let's say I have a random effect intercept. For example: lme4::lmer(yield ~ 1 + (1|Batch)) How is that different than just ordinary regression using ...
3
votes
0answers
81 views

What is the relationship, if any, between Stein's Paradox and linear restrictions in regressions?

Suppose we have $$y = b_1x_1 + b_2x_2 + b_3x_3 + e$$ as our regression model. Setting a linear restriction, say $b_1 + b_2 + b_3 = 0$, allow us to rewrite the model as, $$y = (b_1)(x_1 - x_3) + (...