I have a linear model with response variable $\textbf{y}$ and explanatory variable matrix $\textbf{X}$ for which coefficients $\textbf{b}$ are physically meaningful and worth estimating:

\begin{equation} \textbf{y} = \textbf{X}\textbf{b} + \textbf{e} \end{equation}

However, the relationship between $\textbf{y}$ and $\textbf{X}$ is not strictly linear over the entire domain, but could be better modeled as such within several subgroups $g$ (and coefficients are more meaningful if defined for each subgroup):

\begin{equation} \textbf{y}_g = \textbf{X}_g\textbf{b}_g + \textbf{e}_g \end{equation}

If we arrive at the subgroups through cluster analysis of the explanatory variables $\textbf{X}$ or some prior segregation based on similarity, the subgroup models could increasingly suffer from multicollinearity.

I imagine this is not uncommon in multilevel or hierarchical modeling - if the goal is not just to create a predictive model is there a general approach to parameter estimation in such situations?

  • $\begingroup$ Just to be clear, do you imagine that the slopes for all variables $X$ are allowed to vary across groups $g$? Put differently, are all predictors specified to have random/varying slopes? And how many groups were you thinking of? $\endgroup$ – Erik Ruzek Jan 7 at 0:12
  • $\begingroup$ The physical model specifies that all variables $x$ are multiplied by coefficient $b$ to produce $y$. However, to estimate these coefficients it's not clear that slopes for all variables will vary if the subgroups $g$ become increasingly large relative to the number of samples (currently targeting about 10:1 sample to group ratio). $\endgroup$ – hatmatrix Jan 7 at 0:40
  • 1
    $\begingroup$ Is it that you are interested in a "non-nested" model, in which you have (very) many groups and you want to allow each of these groups to have their own random intercept? See lme4.r-forge.r-project.org/book/Ch2.pdf and vulstats.ucsd.edu/pdf/Gelman.ch-13.more-multilevel-models.pdf. $\endgroup$ – Erik Ruzek Jan 7 at 14:54
  • $\begingroup$ @Erik Ruzek yes that's correct - thanks for the links! $\endgroup$ – hatmatrix Jan 8 at 15:44
  • $\begingroup$ (Correct about the "non-nested" part, but there is no intercept.) $\endgroup$ – hatmatrix Jan 8 at 22:21

Since you say :

However, the relationship between $\textbf{y}$ and $\textbf{X}$ is not strictly linear over the entire domain, but could be better modeled as such within several subgroups $g$ (and coefficients are more meaningful if defined for each subgroup):

it sounds to me very much like a mixed effects model (of which multilevel and hierarchical models are special cases) with random slopes for $\textbf{X}$ in subgroups $g$. This will have the general form:

$$y = \textbf{X}\beta+\textbf{Z}u+e$$

where $\beta$ is a vector of fixed effects, $X$ and $Z$ are model matrices for the fixed effects and random effects respectively and $u$ and $e$ are vectors of random effects such that $E(u) = E(e) = 0$

In R you could fit such a model with, for example:

y = func(y ~ X1 + X2 + (X1 + X2 | g ), ...)

where func will be the relevant function from whatever package you choose, eg lme4 or GLMMAdaptive. Note that some packages, eg nlme use different syntax. This will estimate fixed effects (slopes) and random slopes for X1 and X2 and random intercepts for each group. If you do not want random intercepts - ie. you wish to allow the slopes to vary by group, but all pass through the same point on the y axis, then you would use:

y = func(y ~ X1 + X2 + (X1 + X2 + 0 | g ), ...)
| cite | improve this answer | |
  • $\begingroup$ Sorry if it wasn't clear but the slopes are different but that is not to distinguish slopes from intercepts - there are no intercepts in the physical relationship between $\mathbf{X}$ and $\mathbf{y}$ $\endgroup$ – hatmatrix Jan 7 at 23:05
  • 3
    $\begingroup$ That's exactly why I proposes the 2nd model. If that s $\endgroup$ – Robert Long Jan 8 at 4:04
  • $\begingroup$ Ah, my bad! But isn't this estimation method affected by collinearity? Essentially, within each group $g$ the samples become more similar and therefore the variables become more collinear. $\endgroup$ – hatmatrix Jan 8 at 15:45
  • 2
    $\begingroup$ One of the main reasons for using mixed effects models in the first place is to account for correlations within each group - usually you would fit random intercepts to deal with that. Even if you don't expect the intercepts to vary, measurement error, and natural variation might mean that there is meaningful variance in the intercepts. If the covariates are collinear within each group then you would indeed have a problem, but you can easily check that by subsetting the data and checking the covariance matrix for the groups, but I wouldn't normally expect that to be an issue. $\endgroup$ – Robert Long Jan 8 at 16:18
  • $\begingroup$ I think mixed effects models may be the general direction I want to head into then (as a superclass of hierarchical models), but fundamentally this doesn't address my original question of how to address multicollinearity that arises within each group... $\endgroup$ – hatmatrix Jan 8 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.