Is there a generalization of the “within” transformation to partial pooling?

The random effects estimator for panel data can be cast as a penalized verion of the "fixed-effects" estimator from econometrics. In both cases, the model is $$y_{it} = \mathbf{D}_{i}\alpha_i + \mathbf{X}_{it}\beta + \epsilon$$

Where $\mathbf{D}_{i}$ is a matrix of dummies indicating the cross-sectional unit, and $\alpha_i$ are the respective intercepts.

The random effects estimator used in mixed models is equivalent to ridge regression: $$\hat\alpha = \left(\mathbf{D}^T\mathbf{D}+ \lambda\mathbf{I}\right)^{-1}\mathbf{D}^Ty$$

(ignoring the X's). Obviously the smoothing parameter needs to be selected. Ridge tends to use cross-validation, but likelihood-based methods also exist. These are implemented for example in mgcv, using the syntax

m <- gam(y ~ s(id, bs = "re"), method = "REML")

If I were to specify the smoothing parameter to zero via

m <- gam(y ~ s(id, bs = "re", sp = 0), method = "REML")

then the result would be identical to an OLS regression on the dummy variables, which is to say that $\lambda = 0$.

Now, fixed effects regressions in econometrics are commonly fitted using the "within" transformation (or more generally, the method of alternating projections). Basically, the data are de-meaned: $$y_{it} - \bar{y}_i = \left(\mathbf{D}_{i} - \bar{\mathbf{D}}_{i}\right)\alpha_i + \left(\mathbf{X}_{it} - \bar{\mathbf{X}}_{i}\right)\beta + \epsilon$$

The individual intercepts disappear, and the model can be fit by OLS. This is much simpler to compute than approaches that actually evaluate $\left(\mathbf{D}^T\mathbf{D}+ \lambda\mathbf{I}\right)^{-1}$, because the dimension of that matrix tends to be huge.

So here are my questions.

First: For a given smoothing parameter $\lambda > 0$, is there a faster way to compute $[\hat\alpha, \hat\beta]$ than penalized OLS?? I'd like to avoid the matrix inversion especially.

Second: For a given $\lambda$, is there some $f(., \lambda)$ such that $$y_{it} - f(y_{i}, \lambda) = \left(\mathbf{D}_{i} - f(\mathbf{D}_{i}, \lambda)\right)\alpha_i + \left(\mathbf{X}_{it} - f(\mathbf{X}_{i}, \lambda)\right)\beta + \epsilon$$ yields estimates identical to the ridge regression when fit with OLS?

The plm package uses quasi-demeaning as $f(, \theta) = f(\bar{x}_i\theta)$, where $\theta = 1−[\sigma_u^2 /(\sigma_u^2 +T \sigma_e^2 )]^.5$, where the variances are estimated in the first step of FGLS and $T$ is the number of time periods. But what if I have an exogenously-specified $\lambda$? Is there a one-to-one relationship between $\lambda$ and $\theta$?

First answer: Fastest way might be either: method of alternating projections to find $\hat{\beta}$ and then use the group means to find $\hat{\alpha}$ or you can use an iterative solver like gradient descent / stochastic gradient descent, conjugate gradient solvers, or most suited for your problem a sparse conjugate gradient solver such as LSMR or LSQR for each of the systems of equations.
• Thanks for the answer. I don't actually want fixed effects. I want random effects. Basically I want to take a panel data model, and penalize the whole thing by a given $\lambda$, without losing the computation convenience of the within transformation. Then I'd select $\lambda$ by cross-validation. Generally $\lambda = 0$ is not optimal for prediction, though obvs you've got a different story if you care about inference. – generic_user Nov 8 '17 at 19:57
• If you want a Tikhonov regularized random effects estimator the issue will be that it does not have a true intercept ($1 - \theta_{i}$). Ridge regression uses $\ell_{2}$ penalty which is scale dependent and thus it the model uses the standardized predictors. In this form, a true intercept is needed to get the rescaled coefficients to match the original model (divide by standard deviation of the predictor). I haven't found a clear way to do Tikhonov regularized random effects because of that. Furthermore, should one penalize the random effects intercept or not? I would like to know that myself. – José Bayoán Santiago Calderón Nov 8 '17 at 20:42