I am struck with seemingly easy problem, but I haven't found a suitable solution for several weeks now.
I have quite a lot of poll/survey data (tens of thousands of respondents, say 50k per dataset), coming from something I hope is called complexly designed survey with weights, stratification, specific routing and so on. For each respondents, there are hundreds of variables such as demographics (age, region...) and then mostly binary (at most, categorical) variables.
I come more from computer science/machine learning background and I had to learn a lot about classical survey statistics and methodology. Now I want to apply classical machine learning to those data (e.g. predicting some missing values for subset of respondents - basically classification task). But, hold and behold, I cannot find a suitable way how to do that. How should I incorporate those stratas, weights or routing (like: if question 1 answered with option 2, ask question 3, otherwise skip it)?
Simply applying my models (trees, logistic regression, SVM, XGBoost...) seems dangerous (and, they fail in most cases), since they usually assume data are coming from simple random sample or iid.
A lot of methods at least have weights, but it doesn't help much. Furthermore, it is unclear how I should I combine imbalanced classes and weights given by survey definition together, not talking about those stratification stuff. Furthermore, result models should be well calibrated - the predicted distribution should be very close to the original one. Good performance of prediction isn't the only criteria here. I changed the optimisation metric to take into account this as well (such as distance of predicted distribution from the true distribution + accuracy/MCC) and it helped in some cases, why crippling the performance in others.
Is there some canonical way how to deal with this problem? It seems as a heavily underappreciated area of research for me. IMO many surveys could benefit from ML's power, but there are no sources. Like these are two worlds not interacting with each other.
What I have found so far:
For instance, I still know of only one paper (Toth & Eltinge, 2011) on how to do regression trees when your data come from a complex sample survey.
In a recent meta-analysis of 150 sampled research papers analyzing several surveys with complex sampling designs, it is found that analytic errors caused by ignorance or incorrect use of the complex sample design features were frequent.
Related CV questions, but none of them contains any usable answer how to approach this (either no answer, not what I ask for, or present misleading recommendations):