2
$\begingroup$

I have a question about one of the papers I'm reviewing. The authors used a piecewise growth curve model to estimate trajectories of key variables before and after the middle to high school transition. Their sample is about 2,000 kids pulled from 11 middle schools in one city that were surveyed in both the fall and spring of 7th, 8th, 9th, and 10th grade. In other words, four time points before and after the high school transition. Additionally, the data are obviously nested (observation, individual, school)

They pointed out that the best way to account for the nested structure of the data would be to cluster by school feeder patterns (where kids went to for both middle and high school), but there was significant missing data (about 20%) in the feeder patterns. The authors opted to not cluster by school at all because of the resulting sample size concerns.

Is this the best choice they could have made, or should they have at least nested the students within middle schools? If this is the best choice, how will that affect the results? I worry that not clustering will drastically skew the data. Would love to know your thoughts.

Improve this question
$\endgroup$
3
  • $\begingroup$ I assume they were trying to fit a model with a level-2 random intercept for each student, and they wanted to add a level-3 random intercept of some sort. When you say "cluster by school feeder patterns", what's the nature of the random intercept? Do you just mean that different middle and high schools have their own ID numbers, or something more complex? $\endgroup$
    – Weiwen Ng
    Commented Jan 9, 2018 at 14:34
  • $\begingroup$ Seems so. Seems they nested time (lvl 1) in individual (lvl 2), then used school-level covariates at the individual level (lvl 2). By feeder pattern, they would probably mean that level 3 would've been an ID for each middle school/high school pair. "Because we did not have valid feeder patterns for approximately 20% (n = 446) of the sample and because an additional 25% (n = 492) did not attend their designated feeder high school (resulting in a large number of feeder patterns with n < 5), we did not cluster the data by school feeder pattern in order to retain a larger sample size." $\endgroup$
    – mrjaws
    Commented Jan 9, 2018 at 16:02
  • $\begingroup$ @Weiwen-Ng I forgot to tag you in the post. $\endgroup$
    – mrjaws
    Commented Jan 9, 2018 at 17:45

1 Answer 1

Reset to default
1
$\begingroup$

Where there's some sort of nesting, you generally need to account for it in your model. The issue of observations nested within individuals is an obvious one. Generally, observations within a cluster will be more similar to each other than to two random observations from different clusters. Ignoring that can produce wrong statistical inference.

I don't work in education statistics, so I don't know the discipline's customary practices. I do work in healthcare statistics, and some things are parallel. At face value, without knowing the content of the actual analysis, it does seem like nesting within an individual school should be accounted for. So, minus points for not doing so.

Without considering the complication of people not attending their designated feeder schools, I would probably have run one complete data-only analysis and estimated the intra-class correlations. I would then have run a sensitivity analysis with some attempt to account for the missing data. Even low ICCs can affect your conclusions, but if their school-level ICCs were indeed very very low, then I might be less unwilling to buy this.

There's likely to be some complication with this whole issue of feeder patterns that I could be missing. If someone didn't attend their designated high school, do we have no idea what school they attended at all? How many high schools are we talking about?

I am not sure that their issue with a large number of sparse feeder patterns is a valid one. In a mixed model, the value of the random intercepts is a combination of the grand mean and the cluster mean. If one cluster has few observations, then its predicted random intercept is mostly going to look like the grand mean - for good and for ill, I know, but in that case, small clusters are now irrelevant if all you you want is inference on the fixed effects.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.