Statistical testing is for making inference from data, it tells you how things are related. The result is something that has a real-world meaning. E.g. how smoking is associated with lung cancer, both in terms of direction and magnitude. It still does not tell you why things happened. To answer why things happened, we need to consider also the interrelationship with other variables and make appropriate adjustments (see Pearl, J. (2003) CAUSALITY: MODELS, REASONING, AND INFERENCE).
Supervised learning is for making predictions, it tells you what will happen. E.g. Given the smoking status of a person, we can predict whether s/he will have lung cancer. In simple cases, it still tells you “how”, for example by looking at the cutoff of smoking status that identified by the algorithm. But more complex models are harder or impossible to interpret (deep learning/boosting with a lot of features).
Unsupervised learning is often used in facilitating the above two.
- For statistical testing, by discovering some unknown underlying subgroups of the data (clustering), we can infer the heterogeneity in the associations between variables. E.g. smoking increases the odds of having lung cancer for subgroup A but not subgroup B.
- For supervised learning, we can create new features to improve prediction accuracy and robustness. E.g. by identifying subgroups (clustering) or combination of features (dimension reduction) that are associated with odds of having lung cancer.
When the number of features/variables gets larger, the difference between statistical testing and supervised learning become more substantial. Statistical testing may not necessarily benefit from this, it depends on for example whether you want to make causal inference by controlling for other factors or identifying heterogeneity in the associations as mentioned above. Supervised learning will perform better if the features are relevant and it will become more like a blackbox.
When the number of sample gets larger, we can get more precise results for statistical testing, more accurate results for supervised learning and more robust results for unsupervised learning. But this depends on the quality of the data. Bad quality data may introduce bias or noise to the results.
Sometimes we want to know “how” and “why” to inform interventional actions, e.g. by identifying that smoking causes lung cancer, policy can be made to deal with that. Sometimes we want to know “what” to inform decision-making, e.g. finding out who is likely to have lung cancer and give them early treatments. There is a special issue published on Science about prediction and its limits (http://science.sciencemag.org/content/355/6324/468). “Success seems to be achieved most consistently when questions are tackled in multidisciplinary efforts that join human understanding of context with algorithmic capacity to handle terabytes of data.” In my opinion, for example, knowledge discovered using hypothesis testing can help supervised learning by informing us what data/features we should collect in the first place. On the other hand, supervised learning can help generating hypotheses by informing which variables