9
$\begingroup$

I have worked a lot with various nationally representative data. These data sources have a complex survey design, so the analysis requires the specification of stratification and weight variables. Among the data sources that are within my area of study, machine learning tools have not been applied to them. One obvious reason is that machine learning methods (currently) do not take into account weight and stratification variables.

The goal of the weighted / stratified analyses is to obtain adjusted population estimates, which is different than the goal / purpose of machine learning. What thoughts do people have about using the nationally representative data sources and ignoring the weight and stratification variables? In other words, what would be your thoughts if you reviewing a machine learning study that was used nationally representative data but ignored the weight and stratification variables, assuming that the researcher / author was up-front about this methodological decision and was not making claims of nationally representative results?

Thanks in advance!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ not accounting for the clustering variables means you will be more confident of your results than you should be. not accounting for the weights completely strips the generalizability. if you are not making representative claims, what then are you making claims of? :) $\endgroup$
    – Anthony Damico
    Commented Jun 11, 2014 at 0:38
  • $\begingroup$ @AnthonyDamico, indeed, that makes sense. However, wouldn't the generalizability be equivalent to convenience samples which are ubiquitous in machine learning studies. Not using weights prevents would prevent population estimates to be derived, but won't the correlational structure remain intact? Thanks in advance! $\endgroup$
    – Brian P
    Commented Jun 12, 2014 at 0:11
  • $\begingroup$ the correlational structure is the problem: survey data is sampled in clusters, not by srs. so it is often overcorrelated and variances need to be calculated with that in mind. people create new methods for complex-sample data all the time; look for papers with survey and design-adjusted keywords and figure out what makes sense :) $\endgroup$
    – Anthony Damico
    Commented Jun 12, 2014 at 4:05
  • $\begingroup$ if you have a linearized design, you could switch it over to a replicate-weighted design with survey:::as.svrepdesign and then somehow force your ml algorithm to sample based on the replicate weights alone -- still needs to be run hundreds of times, but it might make dealing with the clusters easier? :) $\endgroup$
    – Anthony Damico
    Commented Jun 12, 2014 at 4:07
  • $\begingroup$ @AnthonyDamico, Thanks for the clarification on the issue of the correlational structure -- that is a helpful perspective. For some relationships, the clustering will most certainly be a problem. Assume that I ignored the issue of clustering in a ML approach but subsequently explored some of the correlational structures with and without the weight and stratification variables using more common regression techniques. Of course, direct comparisons couldn't be made, but it might offer some insight into the extent to which clustering affects the correlational structure. Any thoughts on that? $\endgroup$
    – Brian P
    Commented Jun 12, 2014 at 15:35

1 Answer 1

Reset to default
2
$\begingroup$

I work for a health care company on our member satisfaction team where weights are constantly applied to match the sample to the populations of our service regions. This is very important for interpretable modeling that aims to explain magnitude of relationships between variables. We also use a lot of ML for other tasks, but it seems like you may be wondering if this is important when using machine learning for prediction.

As you hinted most machine learning techniques were not developed for the purpose of explaining relationships, but for predictive purposes. While a representative sample is important, it may not be critical..until your performance tanks.

If algorithms have sufficient samples to learn respondent types, they will be able to predict new respondents' class (classification) / value (regression) well. For example if you had a data set with 4 variables, height, weight, sex, and age, your algorithm of choice will learn certain types of a person based of these characteristics. Say most people in the population are female, 5'4", 35 years old, and 130 pounds (not fact, just roll with it) and we are trying to predict gender. Now say my sample has a low representation of this demographic proportionally, yet still has a high enough number (N) of this type of person. Our model has learned what that type of person looks like though that type of person is not well represented in my sample. When our model sees a new person with those characteristics it will have learned which label (gender) is most associated with said person. If our sample shows that those characteristics are more related to females than males and this matches the population then all is well. The problem arises when the sample's outcome variable does not represent the population by so much that it predicts a different class / value.

So when it comes down to it, testing your predictive ML model on representative data is where you can find out if you have a problem. However, I think it would be fairly rare to sample in such a biased way that prediction would suffer greatly. If accuracy / kappa statistic / AUC is low or RMSE is high when testing then you might want to shave off those people that over-represent demographics of interest given you have enough data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.