What is the difference between data mining and statistical analysis?

Question

What is the difference between data mining and statistical analysis?

For some background, my statistical education has been, I think, rather traditional. A specific question is posited, research is designed, and data are collected and analyzed to offer some insight on that question. As a result, I've always been skeptical of what I considered "data dredging", i.e. looking for patterns in a large dataset and using these patterns to draw conclusions. I tend to associate the latter with data-mining and have always considered this somewhat unprincipled (along with things like algorithmic variable selection routines).

Nonetheless, there is a large and growing literature on data mining. Often, I see this label referring to specific techniques like clustering, tree-based classification, etc. Yet, at least from my perspective, these techniques can be "set loose" on a set of data or used in a structured way to address a question. I'd call the former data mining and the latter statistical analysis.

I work in academic administration and have been asked to do some "data mining" to identify issues and opportunities. Consistent with my background, my first questions were: what do you want to learn and what are the things that you think contribute to issue? From their response, it was clear that me and the person asking the question had different ideas on the nature and value of data mining.

Answer 1

Jerome Friedman wrote a paper a while back: Data Mining and Statistics: What's the Connection?, which I think you'll find interesting.

Data mining was a largely commercial concern and driven by business needs (coupled with the "need" for vendors to sell software and hardware systems to businesses). One thing Friedman noted was that all the "features" being hyped originated outside of statistics -- from algorithms and methods like neural nets to GUI driven data analysis -- and none of the traditional statistical offerings seemed to be a part of any of these systems (regression, hypothesis testing, etc). "Our core methodology has largely been ignored." It was also sold as user driven along the lines of what you noted: here's my data, here's my "business question", give me an answer.

I think Friedman was trying to provoke. He didn't think data mining had serious intellectual underpinnings where methodology was concerned, but that this would change and statisticians ought to play a part rather than ignoring it.

My own impression is that this has more or less happened. The lines have been blurred. Statisticians now publish in data mining journals. Data miners these days seem to have some sort of statistical training. While data mining packages still don't hype generalized linear models, logistic regression is well known among the analysts -- in addition to clustering and neural nets. Optimal experimental design may not be part of the data mining core, but the software can be coaxed to spit out p-values. Progress!

Answer 2

The difference between statistics and data mining is largely a historical one, since they came from different traditions: statistics and computer science. Data mining grew in parallel out of work in the area of artificial intelligence and statistics.

Section 1.4 from Witten & Frank summarizes my viewpoint so I'm going to quote it at length:

What's the difference between machine learning and statistics? Cynics, looking wryly at the explosion of commercial interest (and hype) in this area, equate data mining to statistics plus marketing. In truth, you should not look for a dividing line between machine learning and statistics because there is a continuum--and a multidimensional one at that--of data analysis techniques. Some derive from the skills taught in standard statistics courses, and others are more closely associated with the kind of machine learning that has arisen out of computer science. Historically, the two sides have had rather different traditions. If forced to point to a single difference of emphasis, it might be that statistics has been more concerned with testing hypotheses, whereas machine learning has been more concerned with formulating the process of generalization as a search through possible hypotheses...

In the past, very similar methods have developed in parallel in machine learning and statistics...

But now the two perspectives have converged.

N.B.1 IMO, data mining and machine learning are very closely related terms. In one sense, machine learning techniques are used in data mining. I regularly see these terms as interchangeable, and in so far as they are different, they usually go together. I would suggest looking through "The Two Cultures" paper as well as the other threads from my original question.

N.B.2 The term "data mining" can have a negative connotation when used colloquially to mean letting some algorithm loose on the data without any conceptual understanding. The sense is that data mining will lead to spurious results and over-fitting. I typically avoid using the term when talking to non-experts as a result, and instead use machine learning or statistical learning as a synonym.

Answer 3

Data mining is categorized as either Descriptive or Predictive. Descriptive data mining is to search massive data sets and discover the locations of unexpected structures or relationships, patterns, trends, clusters, and outliers in the data. On the other hand, Predictive is to build models and procedures for regression, classification, pattern recognition, or machine learning tasks, and assess the predictive accuracy of those models and procedures when applied to fresh data.

The mechanism used to search for patterns or structure in high-dimensional data might be manual or automated; searching might require interactively querying a database management system, or it might entail using visualization software to spot anomalies in the data. In machine-learning terms, descriptive data mining is known as unsupervised learning, whereas predictive data mining is known as supervised learning.

Most of the methods used in data mining are related to methods developed in statistics and machine learning. Foremost among those methods are the general topics of regression, classification, clustering, and visualization. Because of the enormous sizes of the data sets, many applications of data mining focus on dimensionality-reduction techniques (e.g., variable selection) and situations in which high-dimensional data are suspected of lying on lower-dimensional hyperplanes. Recent attention has been directed to methods of identifying high-dimensional data lying on nonlinear surfaces or manifolds.

There are also situations in data mining when statistical inference — in its classical sense — either has no meaning or is of dubious validity: the former occurs when we have the entire population to search for answers, and the latter occurs when a data set is a “convenience” sample rather than being a random sample drawn from some large population. When data are collected through time (e.g., retail transactions, stock-market transactions, patient records, weather records), sampling also may not make sense; the time-ordering of the observations is crucial to understanding the phenomenon generating the data, and to treat the observations as independent when they may be highly correlated will provide biased results.

The central components of data mining are — in addition to statistical theory and methods — computing and computational efficiency, automatic data processing, dynamic and interactive data visualization techniques, and algorithm development.

One of the most important issues in data mining is the computational problem of scalability. Algorithms developed for computing standard exploratory and confirmatory statistical methods were designed to be fast and computationally efficient when applied to small and medium-sized data sets; yet, it has been shown that most of these algorithms are not up to the challenge of handling huge data sets. As data sets grow, many existing algorithms demonstrate a tendency to slow down dramatically (or even grind to a halt).

Answer 4

Data mining is statistics, with some minor differences. You can think of it as re-branding statistics, because statisticians are kinda weird.

It is often associated with computational statistics, i.e. only stuff you can do with a computer.

Data miners stole a significant proportion of multivariate statistics and called it their own. Check the table of contents of any 1990s multivariate book and compare it to a new data mining book. Very similar.

Statistics is associated with testing hypotheses and with model building, whereas data mining is more associated with prediction and classification, regardless of whether there is an understandable model.

Answer 5

I previously wrote a post where I made a few observations comparing data mining to psychology. I think these observations may capture some of the differences you are identifying:

1. "Data mining seems more concerned with prediction using observed variables than with understanding the causal system of latent variables; psychology is typically more concerned with the causal system of latent variables.
2. Data mining typically involves massive datasets (e.g. 10,000 + rows) collected for a purpose other than the purpose of the data mining. Psychological datasets are typically small (e.g., less than 1,000 or 100 rows) and collected explicitly to explore a research question.
3. Psychological analysis typically involves testing specific models. Automated model development approaches tend not to be theoretically interesting." - Data Mining and R

Answer 6

I don't think the distinction you make is really related to the difference between data mining and statistical analysis. You are talking about the difference between exploratory analysis and modelling-prediction approach.

I think the tradition of statisic is build with all steps : exploratory analysis, then modeling, then estimation, then testing, then forecasting/infering. Statistician do exploratory analysis to figure out what the data looks like (function summary under R !) I guess datamining is less structured and could be identified with exploratory analysis. However it uses techniques from statistics that are from estimation, forecasting, classification ....