4
$\begingroup$

I have data where counselors are nested inside units which are nested inside areas. I would prefer to use OLS regression rather than multilevel analysis because I only have 7 areas (which I am told is too small a sample size for multilevel models) and because no one in my organization is interested in units (of which there are 70). Formally the nesting violates the assumptions of regression I know which in turn negatively effects the calculated p values. Is there a way to know how serious this impact is, that is if it really invalidates using OLS?

$\endgroup$

2 Answers 2

2
$\begingroup$

One way to interpret a random effect in your model is like assuming those $7$ areas come from some larger population of potential areas to draw future observations from. If you have no interest in other areas, you could model area as a fixed effect. This is far less efficient though, as you'd be estimating $7-1$ offsets from the intercept, instead of a single random variance for 'areas'.

Technically speaking, you could estimate a random variance for as little as $2$ areas. Do you have any particular reason to believe $7$ would be too few? More importantly, if your concern is the validity of $p$-values, then it is irrelevant whether you are interested in units (or areas): You add hierarchy to your model to account for dependency in the data. You don't have to do anything with the estimated variances if you don't want to.

Is there a way to know how serious this impact is, that is if it really invalidates using OLS?

To my knowledge, there are no measures of 'wrongness' of a model that does not account for dependency. One thing you can do is to run both models to see how large the estimated random effects are, and how large the resulting discrepancy in $p$-values is.

$\endgroup$
1
  • 1
    $\begingroup$ The reason I was concerned about only have 7 areas is that one of the articles I read on this topic yesterday said that you need a minimum of 25 of your highest level (in my case area which is level 3) and suggested that if you had less then MLM would not correct the problem with p values. If I understand your response you disagree with that. $\endgroup$
    – user54285
    Nov 19, 2019 at 22:01
0
$\begingroup$

Some thoughts on determining how series the impact of ignoring Clustering can be on OLS estimates

In general, if your data is clustered, e.g. in your case some sampling units are clustered within 7 areas, and you ignore this fact when conducting OLS, your standard errors, confidence intervals and p values will indeed be upwardly biased compared to commonly assumed simple random sampling design.

Heeringa, West & Berglund (2012) provide a more eloquent explanation. In almost all cases, sampling plans that incorporate cluster sampling result in standard errors for survey estimates that are greater than those from a simple random sampling (SRS) of equal size. The SRS variance estimation formulae and approaches incorporated in the standard programs of most statistical software packages no longer apply, because they are based on assumptions of independence of the sample observations and sample observations from within the same cluster generally tend to be correlated (p.28-29).

Without knowing more specific information about your sampling design, the actual sample, and your precise modeling objectives, it is hard to say with more certainty what exact effect ignoring clustering will do in your specific case. However, the following questions can help you better reflect on the effect of clustering on the standard errors and therefore p values in your case.

  1. Are all clusters equal in size? If some clusters are, say, 6,000 units while others are only 200 units, it is easy to see how over-represented individuals from the largest cluster(s) will be. If you want to read more about the effect of unequal clusters, a great example is provided here

  2. Are units selected with equal probability? If in some clusters the probability of selecting certain individuals was much higher than in other clusters, it is reasonable to expect more biased cluster composition in some cases. This will thus partially preclude correct inference.

  3. Is the sample also stratified? For example, stratification by socio-economic status or age is commonly used. This is very important as usually, stratification leads to lower standard errors than SRS. Consider the following example from Heeringa et al (2010, p.34), according to which standard errors of both stratified and clustered design are the closest to the SRS. In contrast, the only-clustered design gives overinflated standard errors compared to the SRS, whereas the only-stratified design gives markedly lower standard errors compared to the SRS. Thus, I would like to emphasize the point that if your design used both stratification and clustering, their combined effect may produce standard errors comparable to those of using SRS alone.

Further thoughts

You only have 7 clusters, and it is generally true that it is a somewhat small number of clusters, which can, thus, lead to biased results. I am not sure how far you would be prepared to go with your modeling but a quick Wikipedia search showed a number of viable solutions to this problem. That is:

One can use a bias-corrected cluster-robust variance matrix, make T-distribution adjustments, or use bootstrap methods with asymptotic refinements, such as the percentile-t or wild bootstrap, that can lead to improved finite sample inference. Cameron, Gelbach and Miller (2008) provide microsimulations for different methods and find that the wild bootstrap performs well in the face of a small number of clusters.

Finally, you may find it useful to determine how similar cases within each cluster are by calculating the intra-class correlation. You would expect that the cases within each cluster are correlated with another much higher than cases across clusters. But how homogenous each group is remains to be seen


References

Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3), 414-427.

Heeringa, S. G., West, B. T., & Berglund, P. A. (2010). Applied survey data analysis. The UK, London: CRC Press.

$\endgroup$
3
  • $\begingroup$ I did not sample. All staff were sent an instrument about half returned useful ones. The areas likely have a similar number of units, but they have very different number of staff as some are rural and some urban. I am not familiar with most of the solutions you mentioned although I have used bootstrapping. To add complexity to this my dependent variable is a 4 point likert scale so I am using HGLM. Having mainly learned linear MLM I am not sure how this changes things. Since I posted this I have learned that some suggest just using OLS with robust errors to deal with clustering which I may do $\endgroup$
    – user54285
    Nov 19, 2019 at 22:53
  • $\begingroup$ @user54285 yes, the fact that your dependent variable is measured on a 4-point Likert scale surely complicates things, because it is ordered categorical rather than continuous. However, If you want to be practical and time is of the essence for you, you may indeed resort to doing just OLS. Indeed, robust standard errors should bring down your standard errors and make them more in line with what you would have expected had you not had any clusters. $\endgroup$ Nov 19, 2019 at 23:05
  • $\begingroup$ @user54285 Many things that I mentioned for you above were primarily to provide some hints as to the factors that could be taken into account when thinking about how negative the effect from clustering might have been in your case $\endgroup$ Nov 19, 2019 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.