Distinguishing between two groups in statistics and machine learning: hypothesis test vs. classification vs. clustering

Question

Assume I have two data groups, labeled A and B (each containing e.g. 200 samples and 1 feature), and I want to know if they are different. I could:

• a) perform a statistical test (e.g. t-test) to see if they are statistically different.

• b) use supervised machine learning (e.g. support vector classifier or random forest classifier). I can train this on a part of my data and verify it on the rest. If the machine learning algorithm classifies the rest correctly afterwards, I can be sure that the samples are differentiable.

• c) use an unsupervised algorithm (e.g. K-Means) and let it divide all data into two samples. I can then check if these two found samples agree with my labels, A and B.

My questions are:

1. How are these three different ways overlapping/exclusive?
2. Are b) and c) useful for any scientific arguments?
3. How could I get a “significance“ for the difference between samples A and B out of methods b) and c)?
4. What would change if the data had multiple features rather than 1 feature?
5. What happens if they contain a different number of samples, e.g. 100 vs 300?

Answer 1

Great question. Anything can be good or bad, useful or not, based on what your goals are (and perhaps on the nature of your situation). For the most part, these methods are designed to satisfy different goals.

• Statistical tests, like the $t$-test allow you to test scientific hypotheses. They are often used for other purposes (because people just aren't familiar with other tools), but generally shouldn't be. If you have an a-priori hypothesis that the two groups have different means on a normally distributed variable, then the $t$-test will let you test that hypothesis and control your long-run type I error rate (although you won't know whether you made a type I error rate in this particular case).
• Classifiers in machine learning, like a SVM, are designed to classify patterns as belonging to one of a known set of classes. The typical situation is that you have some known instances, and you want to train the classifier using them so that it can provide the most accurate classifications in the future when you will have other patterns whose true class is unknown. The emphasis here is on out of sample accuracy; you are not testing any hypothesis. Certainly you hope that the distribution of the predictor variables / features differ between the classes, because otherwise no future classification help will be possible, but you are not trying to assess your belief that the means of Y differ by X. You want to correctly guess X in the future when Y is known.
• Unsupervised learning algorithms, like clustering, are designed to detect or impose structure on a dataset. There are many possible reasons you might want to do this. Sometimes you might expect that there are true, latent groupings in a dataset and want to see if the results of clustering will seem sensible and usable for your purposes. In other cases, you might want to impose a structure on a dataset to enable data reduction. Either way, you are not trying to test a hypothesis about anything, nor are you hoping to be able to accurately predict anything in the future.

With this in mind, lets address your questions:

1. The three methods differ fundamentally in the goals they serve.
2. b and c could be useful in scientific arguments, it depends on the nature of the arguments in question. By far the most common type of research in science is centered on testing hypotheses. However, forming predictive models or detecting latent patters are also possible, legitimate goals.
3. You would not typically try to get 'significance' from methods b or c.
4. Assuming the features are categorical in nature (which I gather is what you have in mind), you can still test hypotheses using a factorial ANOVA. In machine learning there is a subtopic for multi-label classification. There are also methods for multiple membership / overlapping clusters, but these are less common and constitute a much less tractable problem. For an overview of the topic, see Krumpleman, C.S. (2010) Overlapping clustering. Dissertation, UT Austin, Electrical and Computer Engineering (pdf).
5. Generally speaking, all three types of methods have greater difficulty as the number of cases across the categories diverge.

Answer 2

Not going to address clustering because it's been addressed in other answers, but:

In general, the problem of testing whether two samples are meaningfully different is known as two-sample testing.

By doing a $t$-test, you severely limit the kinds of differences that you're looking for (differences in means between normal distributions). There are other tests which can check for more general types of distances: Wilcoxon-Mann-Whitney for stochastic ordering, Kolmogorov-Smirnov for general differences in one dimension, maximum mean discrepancy or the equivalent energy distance for generic differences on arbitrary input spaces, or lots of other choices. Each of these tests is better at detecting certain kinds of differences, and it's sometimes hard to reason about what kinds of differences they're good or bad at detecting, or to interpret the results beyond a $p$ value.

It might be easier to think about some of these issues if you construct a two-sample test out of a classifier, e.g. as recently proposed by Lopez-Paz and Oquab (2017). The procedure is as follows:

• Split your observations $X$ and $Y$ into two parts each, $X_\text{train}$ and $X_\text{test}$, $Y_\text{train}$ and $Y_\text{test}$.
• Train a classifier to distinguish between $X_\text{train}$ and $Y_\text{train}$.
• Apply the output of the classifier to $X_\text{test}$ and $Y_\text{test}$.
• Count up the portion of times its prediction was correct to get $\hat p$. Apply a binomial test to distinguish the null $p = \tfrac12$ from $p \ne \tfrac12$. If $p \ne \tfrac12$, then the two distributions are different.

By inspecting the learned classifier, you may also be able to interpret the differences between the distributions in a semi-meaningful way. By changing the family of classifiers you consider, you can also help guide the test to look for certain kinds of differences.

Note that it's important to do the train-test split: otherwise a classifier that just memorized its inputs would always have perfect discriminability. Increasing the portion of points in the training set gives you more data to learn a good classifier, but less opportunity to be sure that the classification accuracy is really different from chance. This tradeoff is something that is going to vary by problem and classifier family and is not yet well-understood.

Lopez-Paz and Oquab showed good empirical performance of this approach on a few problems. Ramdas et al. (2016) additionally showed that theoretically, a closely related approach is rate-optimal for one specific simple problem. The "right" thing to do in this setting is an area of active research, but this approach is at least reasonable in many settings if you want a little more flexibility and interpretability than just applying some off-the-shelf standard test.

Answer 3

Only approach (a) serves the purpose of testing hypothesis.

In case of using supervised machine learning algorithms (b), they cannot neither prove or disprove hypothesis about distingness of groups. If machine learning algorithm does not classify the groups correctly it may happen because you used "wrong" algorithm for your problem, or you didn't tuned it enough etc. On another hand, you may "torture" the totally "random" data long enough to produce overfitting model that makes good predictions. Yet another problem is when and how would you know that the algorithm makes "good" predictions? Almost never you would aim at 100% classification accuracy, so when would you know that the classification results prove anything?

Clustering algorithms (c) are not designed for supervised learning. They do not aim at recreating the labels, but to group your data in terms of similarities. Now, the results depend on what algorithm you use and what kind of similarities you are looking for. Your data may have different kinds of similarities, you may want to seek for differences between boys and girls, but the algorithm may instead find groups of poor and rich kids, or intelligent and less intelligent, right- and left-handed etc. Not finding the grouping that you intended does not prove that the grouping does not make sense, but only that it found other "meaningful" grouping. As in previous case, the results may depend on the algorithm used and the parameters. Would it suite you if one in ten algorithms/settings found "your" labels? What if it was one in one hundred? How long would you search before stopping? Notice that when using machine learning in vast majority of cases you won't stop after using one algorithm with default settings and the result may depend on the procedure that you used.

Answer 4

1. a) only answers you the question whether the distribution is different, but not how to distinguish them. b) will also find the best value to differentiate between the two distributions. c) will work if the two distributions have some specific properties. For example it will work with normal distribution but not with some two modal distributions, because the method can differentiate two modes of the same group instead of two different groups.

2. c) is not useful for scientific arguments because of two modal distributions. b) could be used for differentiating two distributions, because you can calculate the significance (see 3.) Though I never met it.

3. By bootstrapping. You calculate the model based on random subsamples 1000 times. You get a score, for example the minimum sum of alpha and beta errors. You sort the score ascending. For 5% confidence you choose the 950th value. If this value is lower than 50% (for equal number of points for group A and B) then with 95% confidence you can disregard the null hypothesis that the distributions are the same. The problem is that if the distributions are both normal, have the same mean, but have a different variation then you won't be able to understand that they are different by ML techniques. On the other hand, you can find a test of variation that will be able to distinguish the two distributions. And it could be the other way around that ML will be stronger than a statistical test and will be able to distinguish the distributions.

4. When you have only one feature in ML you need to find only one value to distinguish the distributions. With two features the border can be a sinus and in multi-dimensional space it could be really weird. So it will be much harder to find the right border. On the other hand, additional features bring additional information. So it will generally allow to distinguish the two distributions easier. If both variable are normally distributed then the border is a line.

5. Smaller samples can behave non-normally because the Central Limit Theorem cannot be applied. Bigger sample start to behave more normally because the Central Limit Theorem starts working. For example the mean of both groups will be almost normally distributed if the sample is big enough. But it is usually not 100 vs 300 but 10 observations against 1000 observations. So according to this site the t-test for difference of mean will work irrespective of distribution if the number of observations is larger than 40 and without outliers.

Answer 5

Statistical testing is for making inference from data, it tells you how things are related. The result is something that has a real-world meaning. E.g. how smoking is associated with lung cancer, both in terms of direction and magnitude. It still does not tell you why things happened. To answer why things happened, we need to consider also the interrelationship with other variables and make appropriate adjustments (see Pearl, J. (2003) CAUSALITY: MODELS, REASONING, AND INFERENCE).

Supervised learning is for making predictions, it tells you what will happen. E.g. Given the smoking status of a person, we can predict whether s/he will have lung cancer. In simple cases, it still tells you “how”, for example by looking at the cutoff of smoking status that identified by the algorithm. But more complex models are harder or impossible to interpret (deep learning/boosting with a lot of features).

Unsupervised learning is often used in facilitating the above two.

• For statistical testing, by discovering some unknown underlying subgroups of the data (clustering), we can infer the heterogeneity in the associations between variables. E.g. smoking increases the odds of having lung cancer for subgroup A but not subgroup B.
• For supervised learning, we can create new features to improve prediction accuracy and robustness. E.g. by identifying subgroups (clustering) or combination of features (dimension reduction) that are associated with odds of having lung cancer.

When the number of features/variables gets larger, the difference between statistical testing and supervised learning become more substantial. Statistical testing may not necessarily benefit from this, it depends on for example whether you want to make causal inference by controlling for other factors or identifying heterogeneity in the associations as mentioned above. Supervised learning will perform better if the features are relevant and it will become more like a blackbox.

When the number of sample gets larger, we can get more precise results for statistical testing, more accurate results for supervised learning and more robust results for unsupervised learning. But this depends on the quality of the data. Bad quality data may introduce bias or noise to the results.

Sometimes we want to know “how” and “why” to inform interventional actions, e.g. by identifying that smoking causes lung cancer, policy can be made to deal with that. Sometimes we want to know “what” to inform decision-making, e.g. finding out who is likely to have lung cancer and give them early treatments. There is a special issue published on Science about prediction and its limits (http://science.sciencemag.org/content/355/6324/468). “Success seems to be achieved most consistently when questions are tackled in multidisciplinary efforts that join human understanding of context with algorithmic capacity to handle terabytes of data.” In my opinion, for example, knowledge discovered using hypothesis testing can help supervised learning by informing us what data/features we should collect in the first place. On the other hand, supervised learning can help generating hypotheses by informing which variables