How to estimate parameters in linear mixed models w/ larger number of random effects than observations?

If you have a linear mixed model where

$$y_{n \times 1} = X_{n \times p} \beta_{p \times 1} + Z_{n \times k} \alpha_{k \times 1} + \epsilon_{n \times 1},$$

where $\alpha \sim N(0,\sigma^2I_k)$ and $\epsilon \sim N(0, \tau^2I_n)$, but $k >> n$ (and $p << n$), can you still fit the model in R using the lmer() function in the lme4 package to get the REML estimates of $\sigma^2$ and $\tau^2$? I wonder if the large number of random effects would not work.

If I can use lmer(), then how would the formula look if $k$ is super large? If I can't use lmer(), what would you suggest to get REML estimates of $\sigma^2$ and $\tau^2$?

• If Z is known, you can easily find the ditribution of $Z\alpha$ and this in your noise model, giving rise to a weighted least squares problem. – Yair Daon Sep 25 '17 at 21:48
• I think your $k$ usually should $\le p$ and the column of $Z$ will be less than $X$. I think we estimate fixed effect first then random effects. If there are only random and no fixed effect, why you estimate them then? – Deep North Sep 26 '17 at 5:53
• @Deep North: In genetic models, you sometimes have $k > n$ and the variance components may be the parameter of interest. Note that there are two variance components here. – hiya Sep 26 '17 at 6:00
• I understand the variance compoents for $ZDZ'+R$. but not know why people only want to estiamte rondoma effect without fixed effect (this is infered from $k>>n$ and $p<<n$ therefore $k$ will be much bigger than $p$) – Deep North Sep 26 '17 at 6:09