# Machine learning with weighted / complex survey data

I have worked a lot with various nationally representative data. These data sources have a complex survey design, so the analysis requires the specification of stratification and weight variables. Among the data sources that are within my area of study, machine learning tools have not been applied to them. One obvious reason is that machine learning methods (currently) do not take into account weight and stratification variables.

The goal of the weighted / stratified analyses is to obtain adjusted population estimates, which is different than the goal / purpose of machine learning. What thoughts do people have about using the nationally representative data sources and ignoring the weight and stratification variables? In other words, what would be your thoughts if you reviewing a machine learning study that was used nationally representative data but ignored the weight and stratification variables, assuming that the researcher / author was up-front about this methodological decision and was not making claims of nationally representative results?

• not accounting for the clustering variables means you will be more confident of your results than you should be. not accounting for the weights completely strips the generalizability. if you are not making representative claims, what then are you making claims of? :) – Anthony Damico Jun 11 '14 at 0:38
• @AnthonyDamico, indeed, that makes sense. However, wouldn't the generalizability be equivalent to convenience samples which are ubiquitous in machine learning studies. Not using weights prevents would prevent population estimates to be derived, but won't the correlational structure remain intact? Thanks in advance! – Brian P Jun 12 '14 at 0:11
• the correlational structure is the problem: survey data is sampled in clusters, not by srs. so it is often overcorrelated and variances need to be calculated with that in mind. people create new methods for complex-sample data all the time; look for papers with survey and design-adjusted keywords and figure out what makes sense :) – Anthony Damico Jun 12 '14 at 4:05
• if you have a linearized design, you could switch it over to a replicate-weighted design with survey:::as.svrepdesign and then somehow force your ml algorithm to sample based on the replicate weights alone -- still needs to be run hundreds of times, but it might make dealing with the clusters easier? :) – Anthony Damico Jun 12 '14 at 4:07
• @AnthonyDamico, Thanks for the clarification on the issue of the correlational structure -- that is a helpful perspective. For some relationships, the clustering will most certainly be a problem. Assume that I ignored the issue of clustering in a ML approach but subsequently explored some of the correlational structures with and without the weight and stratification variables using more common regression techniques. Of course, direct comparisons couldn't be made, but it might offer some insight into the extent to which clustering affects the correlational structure. Any thoughts on that? – Brian P Jun 12 '14 at 15:35