Hello fellow StackExchange users,

Preamble: I have been tasked with performing a cluster analysis (or possibly a latent class analysis, as I am pondering) to find non-overlapping groups of like subjects presenting with similar psycho-social traits, as measured in the dataset. There are a priori hypotheses about the number of groups and their characteristics, but the analysis is to be exploratory rather than confirmatory nonetheless. The dataset contains several thousand subjects and about two-dozen variables of interest. So far, so good.

However, there is a catch: the survey comes from a complex multistage stratified design, which obviously violates the SRS assumption and its resulting traditional standard errors. Such designs often require bootstrapping to provide valid inferences. Observation weights have been provided by the architects of the survey.

My question: given that the statistics are exploratory and not confirmatory, and that no formal inferences of any kind are to be produced in this patient-centered analysis (not even precision estimates such as confidence intervals), can the complex survey design be ignored in good faith?

The only caveat may be one of external validity; certain demographic groups have been oversampled. But using the observation weights when calculating the distance/covariance matrices should account for this, no?

Many thanks,


1 Answer 1


There is no good answer to this. Let us consider a big trio of survey statistics: stratification, clustering/multistage selection procedures, and unequal probabilities of selection/sampling weights.

The reason survey statisticians utilize stratification is because they don't want to sample little bit of everything and avoid repeatedly sampling similar units; a perfect stratification is when you find a group of highly similar objects and just take a handful of these. So you should a priori expect the units within the same stratum to be similar; I don't know if you can come up with a measure of distance that accounts for that -- e.g., by discounting the distances of units if they are within the same stratum?

Likewise, the units within the same primary sampling unit may be similar, although that is a parasitic effect for survey estimates that increases variances.

The effect of the sample weights on cluster analysis is totally unclear. The weights are attached to individual units, so how do you come up with a weight for their distance? Survey statisticians sometimes work with pairwise selection probability weights, but this is only relevant if you are trying to estimate a population quantity that is a U-statistic of higher order (Gini index), or involves the second order moments of the design (the variance of Horwitz-Thompson estimator). Distances in cluster analysis are second order moments of the data, so it is not entirely clear whether they should be accompanied with survey weights, and if they should be, how these weights should be expressed.

Oversampling may lead to some strange effects in terms of the sensitivity of your algorithm and the number of detected clusters. If you have few observations with high weights that represent a significant fraction of the population (either because few responded, or by design), your algorithm may miss all of them if it has specification of the smallest number of points it will want to consider as an interesting cluster.

Ideally, you would want to purge the survey design information from your data and analyze the data that have the underlying dependencies only. However, yet another feature of good surveys is that the design information is often related to the outcomes of interest, often in inexplicable ways.

As a bottom line, I am happy that I don't have to do this work :)

BTW, your unequivocal faith in the ability of the bootstrap to fix up anything may not be well justified. I can name 3-5 or so complex survey bootstrap schemes off the top of my head, and another 5-8 if I look up my notes. Shao (1996) gave a great review of the existing resampling methods; nothing absolutely major came up since then.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.