I just watched a video of an interested talk from PyData LA: "Using Simpson’s Paradox to Discover Interesting Patterns in..." - Nazanin Alipourfard, Peter Fennell (https://www.youtube.com/watch?v=Hud8pMbup78). In the talk, the speakers presented a tool they developed to detect confounders by binning explicit features that might act as confounders of trends in other features.

The question that jumped out in my mind is that many features in real world data sets are hidden variables. If I have data that strongly exhibits the "stairstep" pattern typical in the Yule–Simpson effect, but I do not have a second categorical (or binnable) features differentiates the "steps", is there a good clustering technique to recover what I believe to be a hidden variable at work?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.