Is sampling relevant in the time of 'big data'?

Question

Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory.

I've seen this hype around 'Big Data' and can't help wonder that "why" would I want to analyze everything? Wasn't there a reason for "Sampling Theory" to be designed/implemented/invented/discovered? I don't get the point of analyzing the entire 'population' of the dataset. Just because you can do it doesn't mean you should (Stupidity is a privilege but you shouldn't abuse it :)

So my question is this: Is it statistically relevant to analyze the entire data set? The best you could do would be to minimize error if you did sampling. But is the cost of minimizing that error really worth it? Is the "value of information" really worth the effort, time cost etc. that goes in analyzing big data over massively parallel computers?

Even if one analyzes the entire population, the outcome would still be at best a guess with a higher probability of being right. Probably a bit higher than sampling (or would it be a lot more?) Would the insight gained from analyzing the population vs analyzing the sample differ widely?

Or should we accept it as "times have changed"? Sampling as an activity could become less important given enough computational power :)

Note: I'm not trying to start a debate but looking for an answer to understand the why big data does what it does (i.e. analyze everything) and disregard the theory of sampling (or it doesn't?)

Answer 1

In a word, yes. I believe there are still clear situations where sampling is appropriate, within and without the "big data" world, but the nature of big data will certainly change our approach to sampling, and we will use more datasets that are nearly complete representations of the underlying population.

On sampling: Depending on the circumstances it will almost always be clear if sampling is an appropriate thing to do. Sampling is not an inherently beneficial activity; it is just what we do because we need to make tradeoffs on the cost of implementing data collection. We are trying to characterize populations and need to select the appropriate method for gathering and analyzing data about the population. Sampling makes sense when the marginal cost of a method of data collection or data processing is high. Trying to reach 100% of the population is not a good use of resources in that case, because you are often better off addressing things like non-response bias than making tiny improvements in the random sampling error.

How is big data different? "Big data" addresses many of the same questions we've had for ages, but what's "new" is that the data collection happens off an existing, computer-mediated process, so the marginal cost of collecting data is essentially zero. This dramatically reduces our need for sampling.

When will we still use sampling? If your "big data" population is the right population for the problem, then you will only employ sampling in a few cases: the need to run separate experimental groups, or if the sheer volume of data is too large to capture and process (many of us can handle millions of rows of data with ease nowadays, so the boundary here is getting further and further out). If it seems like I'm dismissing your question, it's probably because I've rarely encountered situations where the volume of the data was a concern in either the collection or processing stages, although I know many have

The situation that seems hard to me is when your "big data" population doesn't perfectly represent your target population, so the tradeoffs are more apples to oranges. Say you are a regional transportation planner, and Google has offered to give you access to its Android GPS navigation logs to help you. While the dataset would no doubt be interesting to use, the population would probably be systematically biased against the low-income, the public-transportation users, and the elderly. In such a situation, traditional travel diaries sent to a random household sample, although costlier and smaller in number, could still be the superior method of data collection. But, this is not simply a question of "sampling vs. big data", it's a question of which population combined with the relevant data collection and analysis methods you can apply to that population will best meet your needs.

Answer 2

While there may be hell of a lot of Big Data being produced by the mobile devices and such, there is little usable data in it. If you want to predict the urban travel patterns using foursquare, you may be off by an order of magnitude in estimated flows. Worse, you won't know if you are overestimated or underestimating these flows. You can get an insanely accurate picture of the urban travel patterns of maniacal foursquare users, but unless everyone is required (1) to keep a working smartphone, (2) to run the foursquare app all the time, and (3) to register at any place they stay at for longer than 10 minutes (i.e., get an electronic Census; let libertarians complain about Google and Facebook knowing everything about you), your data will contain unknown biases, and your electronic Deweys will continue to defeat the real-word Trumans (clickable):

_{(source: whatisasurvey.info)}

If anything, I would expect that this piece of history will be repeating itself, and some big "beer+diapers" forecasts produced from Big Data would be overturned by researchers using more rigorous sampling approaches. It is surprising that probability-based surveys remain accurate even despite falling response rates.

Answer 3

Whenever one applies techniques of statistical inference, it is important to be clear as to the population about which one aims to draw conclusions. Even if the data that has been collected is very big, it may still relate only to a small part of the population, and may not be very representative of the whole.

Suppose for example that a company operating in a certain industry has collected 'big data' on its customers in a certain country. If it wants to use that data to draw conclusions about its existing customers in that country, then sampling might not be very relevant. If however it wants to draw conclusions about a larger population - potential as well as existing customers, or customers in another country - then it becomes essential to consider to what extent the customers about whom data has been collected are representative - perhaps in income, age, gender, education, etc - of the larger population.

The time dimension also needs to be considered. If the aim is to use statistical inference to support predictions, then the population must be understood to extend into the future. If so, then again it becomes essential to consider whether the data set, however large, was obtained in circumstances representative of those that may obtain in the future.

Answer 4

From what I've seen of the big data/ML craze, thinking about sampling and the population from which your sample is drawn is just as important as ever--but thought about even less.

I'm "auditing" the Stanford ML class, and thus far we've covered regression and neural networks with nary a mention of population inference. Since this class has been taken by 6 figures-worth of people, there are now an awful lot of people out there who know how to fit data very will without any notion of the idea of a sample.

Answer 5

Yes, sampling is relevant and will remain relevant. Bottom line is that the accuracy of a statistical estimate is generally a function of the sample size, not the population to which we want to generalize. So a mean or an average proportion computed from a sample of 1,000 respondents will yield an estimate of a certain accuracy (with respect to the entire population from which we sampled), regardless of the size of the population (or “how big” the “big data” are are).

Having said that: There are specific issues and challenges that are relevant and should be mentioned:

1. Taking a good probability sample is not always easy. Theoretically, every individual in the population to which we want to generalized (about which we want to make inferences) must have a known probability of being selected; ideally that probability should be the same (equal probability sample or EPSEM – Equal Probability of Selection). That is an important consideration and one should have a clear understanding of how the sampling process will assign selection probabilities to the members of the population to which one wants to generalize. For example, can one derive from Twitter feeds accurate estimates of overall sentiments in the population at large, including those individuals without twitter accounts?
2. Big data may contain very complex details and information; put another way, the issue is not sampling, but (micro-) segmentation, pulling out the right details for a small subset of observations that are relevant. Here the challenge is not sampling, but to identify the specific stratification and segmentation of the big data that yields the most accurate actionable information that can be turned into valuable insights.
3. Another general rule of opinion measurement is that non-sampling errors and biases are usually much bigger than the sampling error and biases. Just because you process 1 hundred gazillion records of respondents expressing an opinion doesn’t make the results more useful if you only have data of a 1000 person subsample, in particular if the questions for the respective survey were not written well and induced bias.
4. Sometimes sampling is required: For example, if one were to build a predictive model from all data, how would one validate it? How would one compare the accuracy of different models? When there are “big data” (very large data repositories) then one can build multiple models and modeling scenarios for different samples, and validate them (try them out) in other independent samples. If one were to build one model for all data – how would one validate it?

You can check out our 'Big Data Revolution' here.

Answer 6

Many big data methods are actually designed around sampling.

The question should be more on the line of:

Shouldn't we use systematic sampling with big data, too?

A lot of the "big data" stuff is still pretty fresh, and sometimes naive. K-means for example can be trivially parallelized, and thus works for "big data" (I'm not going to talk about the results, they are not very meaningful; and probably not very different to those obtained on a sample!). As far as I know this is what the k-means implementation in Mahout does.

However, research is going beyond the naive parallelization (that may still require a large amount of iterations) and tries to do K-means in a fixed number of iterations. Example for this:

• Fast clustering using MapReduce
  Ene, A. and Im, S. and Moseley, B.
  Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, 2011

And guess what, their approach is heavily based on sampling.

Next example: Decision forests. That is essentially: for several samples from the data set, build a decision tree each. Can again be trivially parallelized: put each sample on a separate machine. And again, it is a sampling based approach.

So sampling is one of the key ingredients to big data approaches!

And there is nothing wrong with this.

Answer 7

Cross validation is an specific example of sub-sampling which is quite important in ML/big data. More generally, big data is still usually a sample of a population, as other people here have mentioned.

But, I think OP might be specifically referring to sampling as it applies to a controlled experiments, versus observational data. Usually big data is thought of as the latter, but to me at least there are exceptions. I would think of randomized trials, A/B testing, and multiarmed bandits in e-commerce and social network settings as examples of "sampling in big data settings."

Answer 8

In the areas where Big Data is gaining popularity: Search, Advertising, Recommender Systems like Amazon, Netflix , there is a very Big incentive to explore the entire data set.

The objective of these systems is to tailor recommendations / suggestions to every single member of the population. Also, the number of attributes being studied is enormous. The average web analytics system may measure click-through rate, "thermal tracking" of the "hot areas" in a page, social interactions, etc and weigh these against a large set of predetermined objectives.

More importantly, most of the places where Big Data is now ubiquitous are "online" data streams i.e data is constantly being added / updated. Devising a sampling scheme which covers all these attributes without an inherent bias and still deliver promising results (read better margins) is a challenge.

Sampling still remains highly relevant for surveys, medical trials, A/B testing, quality assurance.

In a nutshell, sampling is very useful when the population to be studied is very large and you are interested in the macroscopic properties of the population. 100% checking (Big Data) is necessary for exploiting the microscopic properties of the system

Hope this helps :)