Valid references on origins of Machine Learning, Statistical Learning and Data Mining

Question

I know it's a rather debated question on Stack Exchange communities but let me explain the points of this question.

I'm writing my capstone on Machine Learning and I need to clarify deeply, giving valid references, the differences among Data Mining, Big Data, Statistical Learning and Machine Learning. As far as I understand (I'm reading The Elements of Statistical Learning, An Introduction to Statistical Learning which are well known books) and based on the questions linked below, those are basically Machine Learning. Correct me if you do not agree with me.

However, I can't find an article or book with a detailed discussion on the origins of those fields and relationship of each other and I can't cite, like, the questions below. They're not considered valid references.

1. What is the difference between data mining, statistics, machine learning and AI?

2. What is the difference between Big Data and Data Mining?

3. difference between data mining and machine learning [closed]

Answer 1

The issues you're noting are definitional ones where standard, widely accepted meanings for each term have yet to be agreed upon -- different authors and practitioners use them differently. I think nearly everyone would agree that there is a high degree of overlap in their use. This is frequently the case during the emergence of relatively new fields. So, 10 or more years ago, data mining was widely considered to be a "bad" thing relative to theoretical, hypothesis-driven standards of research -- the "gold standard." Today, the stigma associated with data mining has been, for the most part, removed in common parlance.

Regrettably, these considerations can devolve into dogmatic, almost religious wars of turf where the contending definitions are a function and by-product of the discipline (the "turf") within which they originate. So, machine learning has largely developed within computer science departments, whose content overlaps with statistics, but it can be treated as a wholly separate discipline from statistics with a separate literature to command. Indeed, many ML practitioners will acknowledge that their exposure and experience evolved without any statistical considerations coming into play whatsoever. A good example of this is Chen and Xie's paper on "divide and conquer" algorithms for massive data -- http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf -- which notes that D&C approaches originated in ML and computer science but without any statistical consideration given to the accuracy of the approximating results. It took statisticians like Chen and Xie to ask and address the concern.

If it's helpful, one way that might reduce some of the confusion is to think of the relationships in Boolean terms. You could even develop a text mining algorithm to show the overlap in Venn diagrams based on term usage from a set of related documents. So, AI is a subset of ML. ML is a subset of computer science that overlaps with statistics. And statistics may (or may not) be a subset of mathematics. Big data is a technical consideration that is largely a computer science concern but it is also a subset of issues having to do with IT hardware, software and data architecture. But big data has impact on statistical analysis insofar as most 20th c approaches to statistical modeling have significant "in-memory" software limitations when the data gets too big. Data mining is an approach to exploratory research that is a subset of methodological and research design issues that overlaps with statistics as well as ML. And so on.

The bottom line is that the more you read on these topics, the closer you will get to arriving at your own understanding and definitions. You may have to get creative in this regard since you are unlikely to find crisp definitions in one, two or even a few sources.

Answer 2

In addition to Mike Hunter's excellent answer (+1), I'd like to point out the envisioned application fields of statistics, contrasted to Machine Learning.

Statistics was historically developed as a tool for informing human decision makers about dependencies in social and physical phenomena. Humans would analyse the statistical findings, consult, debate and, eventually, after a lot of deliberations, make a decision. Statistical analysis comes with a load of "ifs" and "buts" and is just a help to human experts.

Machine learning, on the other hand, evolved from attempts to allow machines make autonomous decisions, ideally without human help at all once the training process has been completed. The internal representation of what the machine has learned is not intended to be human-readable and an analysis of the findings is seldom possible.

As an example, think of economics vs. stock trading. If you were an economist in a national bank, you'd statistically analyse data regarding the country's economy (unemployment, inflation, GDP etc.) and, after a lot of deliberations and consultations with your peers, decide whether to rise the interest rates or not. Statistics is here a tool helping the expert make an informed decision.

At the stock market, in contrast, you want to react as soon as possible, in some cases even within milliseconds, to market changes. You don't have time to ponder and discuss. Either you trust your experience and gut feeling, or you leave the decision to a suitably trained machine.

Another example is traffic: Statistics might be useful to design a road network and traffic signalling, but, once on the road, p-values and confidence intervals are useless. You either drive yourself, based on what you've learned and your experience, or you let a self-driving car drive you. In both cases, you make no use of statistics.