3k views

### Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
3k views

### What are the main theorems in Machine (Deep) Learning?

Al Rahimi has recently given a very provocative talk in NIPS 2017 comparing current Machine Learning to Alchemy. One of his claims is that we need to get back to theoretical developments, to have ...
3k views

Ridge regression estimates parameters $\boldsymbol \beta$ in a linear model $\mathbf y = \mathbf X \boldsymbol \beta$ by $$\hat{\boldsymbol \beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf ... 1answer 2k views ### Is there a connection between empirical Bayes and random effects? I recently happened to read about empirical Bayes (Casella, 1985, An introduction to empirical Bayes data analysis) and it looked a lot like random effects model; in that both have estimates shrunken ... 1answer 2k views ### James-Stein Estimator with unequal variances Every statement I find of the James-Stein estimator assumes that the random variables being estimated have the same (and unit) variance. But all of these examples also mention that the JS estimator ... 1answer 3k views ### Are estimates of regression coefficients uncorrelated? Consider a simple regression (normality not assumed):$$Y_i = a + b X_i + e_i,$$where e_i is with mean 0 and standard deviation \sigma. Are the Least Square Estimates of a and b ... 1answer 2k views ### Estimates of random effects in binomial model (lme4) I'm simulating Bernoulli trials with a random \text{logit}\, \theta \sim {\cal N}(\text{logit}\, \theta_0, 1^2) between groups and then I fit the corresponding model with the ... 1answer 270 views ### What is the rationale behind LARS-OLS hybrid, i.e. using OLS estimate on the variables chosen by LARS? I need some help to understand the relationship between the ranking of the variables from the LARS algorithm and the use of OLS to estimate the final model chosen by the LARS. I understand that the ... 0answers 82 views ### What is the relationship, if any, between Stein's Paradox and linear restrictions in regressions? Suppose we have$$y = b_1x_1 + b_2x_2 + b_3x_3 + e$$as our regression model. Setting a linear restriction, say b_1 + b_2 + b_3 = 0, allow us to rewrite the model as,$$y = (b_1)(x_1 - x_3) + (...
Suppose we observe panel data $$y_{it} = \alpha_i + \beta_i\,t + \epsilon_{it}$$ where $i$ indexes organizations, $t$ is time, and $\epsilon_{it}$ is i.i.d noise. The terms $\alpha_i$ and $\beta_i$ ...