4k views

What are the main theorems in Machine (Deep) Learning?

Al Rahimi has recently given a very provocative talk in NIPS 2017 comparing current Machine Learning to Alchemy. One of his claims is that we need to get back to theoretical developments, to have ...
2k views

James-Stein Estimator with unequal variances

Every statement I find of the James-Stein estimator assumes that the random variables being estimated have the same (and unit) variance. But all of these examples also mention that the JS estimator ...
160 views

Difference between random effect and fixed effect with regularization/prior

Let's say I have a random effect intercept. For example: lme4::lmer(yield ~ 1 + (1|Batch)) How is that different than just ordinary regression using ...
4k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
403 views

Ridge regression / regularization approach to hierarchical model

Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...
3k views

Are estimates of regression coefficients uncorrelated?

Consider a simple regression (normality not assumed): $$Y_i = a + b X_i + e_i,$$ where $e_i$ is with mean $0$ and standard deviation $\sigma$. Are the Least Square Estimates of $a$ and $b$ ...
284 views

What is the rationale behind LARS-OLS hybrid, i.e. using OLS estimate on the variables chosen by LARS?

I need some help to understand the relationship between the ranking of the variables from the LARS algorithm and the use of OLS to estimate the final model chosen by the LARS. I understand that the ...
2k views

Is there a connection between empirical Bayes and random effects?

I recently happened to read about empirical Bayes (Casella, 1985, An introduction to empirical Bayes data analysis) and it looked a lot like random effects model; in that both have estimates shrunken ...
I'm simulating Bernoulli trials with a random $\text{logit}\, \theta \sim {\cal N}(\text{logit}\, \theta_0, 1^2)$ between groups and then I fit the corresponding model with the ...