29
$\begingroup$

Assume I have two data groups, labeled A and B (each containing e.g. 200 samples and 1 feature), and I want to know if they are different. I could:

  • a) perform a statistical test (e.g. t-test) to see if they are statistically different.

  • b) use supervised machine learning (e.g. support vector classifier or random forest classifier). I can train this on a part of my data and verify it on the rest. If the machine learning algorithm classifies the rest correctly afterwards, I can be sure that the samples are differentiable.

  • c) use an unsupervised algorithm (e.g. K-Means) and let it divide all data into two samples. I can then check if these two found samples agree with my labels, A and B.

My questions are:

  1. How are these three different ways overlapping/exclusive?
  2. Are b) and c) useful for any scientific arguments?
  3. How could I get a “significance“ for the difference between samples A and B out of methods b) and c)?
  4. What would change if the data had multiple features rather than 1 feature?
  5. What happens if they contain a different number of samples, e.g. 100 vs 300?
$\endgroup$
  • 3
    $\begingroup$ I would say that the difference between (a) and (b) is that statistical tests focus on whether there is a difference, whereas classification methods focus on the size of this difference. When you use random forest, you want to know the cross-validated accuracy; perhaps it's 78%. That's the number you are interested in, and not in making a statement that it's not equal to 50%. $\endgroup$ – amoeba Feb 18 '17 at 23:53
  • 4
    $\begingroup$ IMHO situations where a/b/c make sense differ in the typical signal-to-noise ratio, and it's increasing from (a) to (b) to (c). In the typical application of a t-test, there is high noise; we want to show that groups are not equal. If the groups are obviously not equal (less noise), we don't really need a test anymore; instead we want to quantify how different the groups are, and out-of-sample classification accuracy can help here. If there is even less noise and classification accuracy is ~100%, we can ask if groups are so super-distinct that it can be picked up by a clustering algorithm. $\endgroup$ – amoeba Feb 19 '17 at 0:03
  • 1
    $\begingroup$ @amoeba, I'm addressing to you because you have been editing the question title etc. I would ask you to consider it one more time. "Classification" and "clustering": are not in the (sole) possession of machine learning. These tasks appeared and were routinely done in statistics / data analysis before the first m. learner was born. It is only true for few recent techniques like SVM which evolved due to and within ML. It is improper to associate classification/clustering/trees with ML only. ML, however, differs from statistical data analysis in the respect that it uses train/test heavily. $\endgroup$ – ttnphns Feb 19 '17 at 15:35
  • $\begingroup$ @ttnphns Sure, that's all correct, but the original title was "Distinguishing samples: Machine Learning vs. Statistical tests (e.g. t-test)" and I just wanted to add some precision to it because the question is really asking about t-test vs classification vs clustering (for this particular scientific purpose). I will think about possible improvements to the title. $\endgroup$ – amoeba Feb 19 '17 at 16:45
  • $\begingroup$ @ttnphns I edited the title, see if you like it more. $\endgroup$ – amoeba Feb 21 '17 at 10:20
15
$\begingroup$

Great question. Anything can be good or bad, useful or not, based on what your goals are (and perhaps on the nature of your situation). For the most part, these methods are designed to satisfy different goals.

  • Statistical tests, like the $t$-test allow you to test scientific hypotheses. They are often used for other purposes (because people just aren't familiar with other tools), but generally shouldn't be. If you have an a-priori hypothesis that the two groups have different means on a normally distributed variable, then the $t$-test will let you test that hypothesis and control your long-run type I error rate (although you won't know whether you made a type I error rate in this particular case).
  • Classifiers in machine learning, like a SVM, are designed to classify patterns as belonging to one of a known set of classes. The typical situation is that you have some known instances, and you want to train the classifier using them so that it can provide the most accurate classifications in the future when you will have other patterns whose true class is unknown. The emphasis here is on out of sample accuracy; you are not testing any hypothesis. Certainly you hope that the distribution of the predictor variables / features differ between the classes, because otherwise no future classification help will be possible, but you are not trying to assess your belief that the means of Y differ by X. You want to correctly guess X in the future when Y is known.
  • Unsupervised learning algorithms, like clustering, are designed to detect or impose structure on a dataset. There are many possible reasons you might want to do this. Sometimes you might expect that there are true, latent groupings in a dataset and want to see if the results of clustering will seem sensible and usable for your purposes. In other cases, you might want to impose a structure on a dataset to enable data reduction. Either way, you are not trying to test a hypothesis about anything, nor are you hoping to be able to accurately predict anything in the future.

With this in mind, lets address your questions:

  1. The three methods differ fundamentally in the goals they serve.
  2. b and c could be useful in scientific arguments, it depends on the nature of the arguments in question. By far the most common type of research in science is centered on testing hypotheses. However, forming predictive models or detecting latent patters are also possible, legitimate goals.
  3. You would not typically try to get 'significance' from methods b or c.
  4. Assuming the features are categorical in nature (which I gather is what you have in mind), you can still test hypotheses using a factorial ANOVA. In machine learning there is a subtopic for multi-label classification. There are also methods for multiple membership / overlapping clusters, but these are less common and constitute a much less tractable problem. For an overview of the topic, see Krumpleman, C.S. (2010) Overlapping clustering. Dissertation, UT Austin, Electrical and Computer Engineering (pdf).
  5. Generally speaking, all three types of methods have greater difficulty as the number of cases across the categories diverge.
$\endgroup$
  • 2
    $\begingroup$ Re #4: I think you misunderstand the word "feature" in the OP. In machine learning, "feature" simply means a variable. So "multiple features" means that one would use a multivariate version of a t-test (such as Hotelling's T), and not a factorial ANOVA. $\endgroup$ – amoeba Feb 18 '17 at 23:50
11
$\begingroup$

Not going to address clustering because it's been addressed in other answers, but:

In general, the problem of testing whether two samples are meaningfully different is known as two-sample testing.

By doing a $t$-test, you severely limit the kinds of differences that you're looking for (differences in means between normal distributions). There are other tests which can check for more general types of distances: Wilcoxon-Mann-Whitney for stochastic ordering, Kolmogorov-Smirnov for general differences in one dimension, maximum mean discrepancy or the equivalent energy distance for generic differences on arbitrary input spaces, or lots of other choices. Each of these tests is better at detecting certain kinds of differences, and it's sometimes hard to reason about what kinds of differences they're good or bad at detecting, or to interpret the results beyond a $p$ value.

It might be easier to think about some of these issues if you construct a two-sample test out of a classifier, e.g. as recently proposed by Lopez-Paz and Oquab (2017). The procedure is as follows:

  • Split your observations $X$ and $Y$ into two parts each, $X_\text{train}$ and $X_\text{test}$, $Y_\text{train}$ and $Y_\text{test}$.
  • Train a classifier to distinguish between $X_\text{train}$ and $Y_\text{train}$.
  • Apply the output of the classifier to $X_\text{test}$ and $Y_\text{test}$.
  • Count up the portion of times its prediction was correct to get $\hat p$. Apply a binomial test to distinguish the null $p = \tfrac12$ from $p \ne \tfrac12$. If $p \ne \tfrac12$, then the two distributions are different.

By inspecting the learned classifier, you may also be able to interpret the differences between the distributions in a semi-meaningful way. By changing the family of classifiers you consider, you can also help guide the test to look for certain kinds of differences.

Note that it's important to do the train-test split: otherwise a classifier that just memorized its inputs would always have perfect discriminability. Increasing the portion of points in the training set gives you more data to learn a good classifier, but less opportunity to be sure that the classification accuracy is really different from chance. This tradeoff is something that is going to vary by problem and classifier family and is not yet well-understood.

Lopez-Paz and Oquab showed good empirical performance of this approach on a few problems. Ramdas et al. (2016) additionally showed that theoretically, a closely related approach is rate-optimal for one specific simple problem. The "right" thing to do in this setting is an area of active research, but this approach is at least reasonable in many settings if you want a little more flexibility and interpretability than just applying some off-the-shelf standard test.

$\endgroup$
  • $\begingroup$ (+1) Also, nested cross-validation is the way to go, imo. Then test the performance estimate you get in the outer resampling loop against a no-information model performance. If significantly higher than expected by random chance, then your data are somewhat discriminated. $\endgroup$ – Firebug Feb 19 '17 at 14:12
  • $\begingroup$ @Firebug What do you mean by a "no-information model performance"? I don't quite get your proposed procedure. $\endgroup$ – Dougal Feb 19 '17 at 14:14
  • 2
    $\begingroup$ @Firebug An important but subtle caveat is that if classification accuracy is estimated via CV then one can't use a binomial test. $\endgroup$ – amoeba Feb 19 '17 at 14:49
  • 2
    $\begingroup$ @Firebug My point still remains: you cannot apply any kind of one-sample test to AUCs from different folds and in particular from repeated CV because these estimates are not independent. This is a well-known problem. $\endgroup$ – amoeba Feb 19 '17 at 16:48
  • 2
    $\begingroup$ Re permutation testing: I have done it myself. You run CV to obtain an estimate of performance, then shuffle the labels and run the whole CV pipeline again (and do this shuffling 100 or 1000 times to obtain the null distribution). This does tend to take an awful lot of time though. Cc to @Firebug. $\endgroup$ – amoeba Feb 19 '17 at 17:21
3
$\begingroup$

Only approach (a) serves the purpose of testing hypothesis.

In case of using supervised machine learning algorithms (b), they cannot neither prove or disprove hypothesis about distingness of groups. If machine learning algorithm does not classify the groups correctly it may happen because you used "wrong" algorithm for your problem, or you didn't tuned it enough etc. On another hand, you may "torture" the totally "random" data long enough to produce overfitting model that makes good predictions. Yet another problem is when and how would you know that the algorithm makes "good" predictions? Almost never you would aim at 100% classification accuracy, so when would you know that the classification results prove anything?

Clustering algorithms (c) are not designed for supervised learning. They do not aim at recreating the labels, but to group your data in terms of similarities. Now, the results depend on what algorithm you use and what kind of similarities you are looking for. Your data may have different kinds of similarities, you may want to seek for differences between boys and girls, but the algorithm may instead find groups of poor and rich kids, or intelligent and less intelligent, right- and left-handed etc. Not finding the grouping that you intended does not prove that the grouping does not make sense, but only that it found other "meaningful" grouping. As in previous case, the results may depend on the algorithm used and the parameters. Would it suite you if one in ten algorithms/settings found "your" labels? What if it was one in one hundred? How long would you search before stopping? Notice that when using machine learning in vast majority of cases you won't stop after using one algorithm with default settings and the result may depend on the procedure that you used.

$\endgroup$
  • 2
    $\begingroup$ For approach (b): you can construct a hypothesis test as noted in my answer to solve the problem of knowing whether your result is meaningful (though this is surely in a naive use of classifiers for this problem). Note that any given hypothesis test might also fail to reject the null because it is the "wrong" test for your problem, in exactly the same kind of ways as would happen with a classifier; the problem of overfitting is fully addressed by data splitting. $\endgroup$ – Dougal Feb 18 '17 at 20:30
  • $\begingroup$ @Dougal good points (+1) but my main argument is that by using machine learning the outcome depends on how do you choose the algorithm, use it, test it and assess the results; so the outcome of such test to great extent depends on your actions. This means that potentially two different statisticians could get different results while using this method. On another hand, with traditional hypothesis tests it depends only on the choice of the test. $\endgroup$ – Tim Feb 18 '17 at 23:05
  • $\begingroup$ Moreover, there is no single "correct" way to apply it and you can easily manipulate it (purposefully or not) to obtain expected results. $\endgroup$ – Tim Feb 18 '17 at 23:08
  • $\begingroup$ If you're just trying a million things until you get the desired outcome, yes. If you do proper data splitting including from your own analysis, then the test procedure is perfectly valid. Having more options allows you greater power in situations where you know how to exploit it, but yes allows for more opportunities to cheat (inadvertently or otherwise) if you're not being careful. $\endgroup$ – Dougal Feb 18 '17 at 23:11
  • $\begingroup$ @Dougal yes, but the outcome depends also on the procedure used for cross-validation and the splitting itself (e.g. size of test group). So at each step the outcome depends on your procedure. Moreover, it is your decision when you stop learning and trying to achieve better results (single algorithm with default settings, vs single algorithms and tuning the parameters, vs multiple algorithms - how many?). This procedure could possibly need some corrections for multiple testing (but what exactly?) to account for the used procedure - good results on single algorithm with default settings seem to $\endgroup$ – Tim Feb 18 '17 at 23:25
2
$\begingroup$
  1. a) only answers you the question whether the distribution is different, but not how to distinguish them. b) will also find the best value to differentiate between the two distributions. c) will work if the two distributions have some specific properties. For example it will work with normal distribution but not with some two modal distributions, because the method can differentiate two modes of the same group instead of two different groups.

  2. c) is not useful for scientific arguments because of two modal distributions. b) could be used for differentiating two distributions, because you can calculate the significance (see 3.) Though I never met it.

  3. By bootstrapping. You calculate the model based on random subsamples 1000 times. You get a score, for example the minimum sum of alpha and beta errors. You sort the score ascending. For 5% confidence you choose the 950th value. If this value is lower than 50% (for equal number of points for group A and B) then with 95% confidence you can disregard the null hypothesis that the distributions are the same. The problem is that if the distributions are both normal, have the same mean, but have a different variation then you won't be able to understand that they are different by ML techniques. On the other hand, you can find a test of variation that will be able to distinguish the two distributions. And it could be the other way around that ML will be stronger than a statistical test and will be able to distinguish the distributions.

  4. When you have only one feature in ML you need to find only one value to distinguish the distributions. With two features the border can be a sinus and in multi-dimensional space it could be really weird. So it will be much harder to find the right border. On the other hand, additional features bring additional information. So it will generally allow to distinguish the two distributions easier. If both variable are normally distributed then the border is a line.

  5. Smaller samples can behave non-normally because the Central Limit Theorem cannot be applied. Bigger sample start to behave more normally because the Central Limit Theorem starts working. For example the mean of both groups will be almost normally distributed if the sample is big enough. But it is usually not 100 vs 300 but 10 observations against 1000 observations. So according to this site the t-test for difference of mean will work irrespective of distribution if the number of observations is larger than 40 and without outliers.

$\endgroup$
0
$\begingroup$

Statistical testing is for making inference from data, it tells you how things are related. The result is something that has a real-world meaning. E.g. how smoking is associated with lung cancer, both in terms of direction and magnitude. It still does not tell you why things happened. To answer why things happened, we need to consider also the interrelationship with other variables and make appropriate adjustments (see Pearl, J. (2003) CAUSALITY: MODELS, REASONING, AND INFERENCE).

Supervised learning is for making predictions, it tells you what will happen. E.g. Given the smoking status of a person, we can predict whether s/he will have lung cancer. In simple cases, it still tells you “how”, for example by looking at the cutoff of smoking status that identified by the algorithm. But more complex models are harder or impossible to interpret (deep learning/boosting with a lot of features).

Unsupervised learning is often used in facilitating the above two.

  • For statistical testing, by discovering some unknown underlying subgroups of the data (clustering), we can infer the heterogeneity in the associations between variables. E.g. smoking increases the odds of having lung cancer for subgroup A but not subgroup B.
  • For supervised learning, we can create new features to improve prediction accuracy and robustness. E.g. by identifying subgroups (clustering) or combination of features (dimension reduction) that are associated with odds of having lung cancer.

When the number of features/variables gets larger, the difference between statistical testing and supervised learning become more substantial. Statistical testing may not necessarily benefit from this, it depends on for example whether you want to make causal inference by controlling for other factors or identifying heterogeneity in the associations as mentioned above. Supervised learning will perform better if the features are relevant and it will become more like a blackbox.

When the number of sample gets larger, we can get more precise results for statistical testing, more accurate results for supervised learning and more robust results for unsupervised learning. But this depends on the quality of the data. Bad quality data may introduce bias or noise to the results.

Sometimes we want to know “how” and “why” to inform interventional actions, e.g. by identifying that smoking causes lung cancer, policy can be made to deal with that. Sometimes we want to know “what” to inform decision-making, e.g. finding out who is likely to have lung cancer and give them early treatments. There is a special issue published on Science about prediction and its limits (http://science.sciencemag.org/content/355/6324/468). “Success seems to be achieved most consistently when questions are tackled in multidisciplinary efforts that join human understanding of context with algorithmic capacity to handle terabytes of data.” In my opinion, for example, knowledge discovered using hypothesis testing can help supervised learning by informing us what data/features we should collect in the first place. On the other hand, supervised learning can help generating hypotheses by informing which variables

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.