If you do a hypothesis to test the effectiveness of a treatment, or a marketing campaign, you want to be sure that the two groups are comparable.

You can compare some relevant quantities between the two groups. Even if these do not differ significantly, there may still be hidden differences between the two groups.

Suppose we compare costumer spending. The test group are costumers whose names start with the letter a, b, c, d, or e. The time and amount at check out are stored.

Can machine learning play a role in finding (statistically significant) differences between the groups?

Are there any references on this kind of use of machine learning?

  • $\begingroup$ Are you asking about a situation where you have a treated and a control group sample and you are looking to understand if they were statistically similar enough pre-marketing to then do a hypothesis test regarding difference "due" to the marketing intervention? $\endgroup$
    – B_Miner
    Jul 25, 2014 at 12:52

1 Answer 1


If you are interested in machine learning for this problem instead of targeted traditional statistical inference you must be interested in comparing entire distributions rather than just the mean or quantiles. That being the case, consider the Kolmogorov-Smirnov two-sample test. If you need to adjust for other subject characteristics in comparing the groups, consider using traditional methods that attempt to isolate the group effect from the other effects, e.g., analysis of covariance or propensity score adjustment.

  • 1
    $\begingroup$ +1. Statistics has a lot of excellent machinery to offer for these situations. I would shy away from approaching these types of problems from a ML angle. $\endgroup$ Jul 25, 2014 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.