# What exactly is Big Data?

I have been asked on several occasions the question:

What is Big-Data?

Both by students and my relatives that are picking up the buzz around statistics and ML.

I found this CV-post. And I feel that I agree with the only answer there.

The Wikipedia page also has some comments on it, but I am not sure if I really agree with everything there.

EDIT: (I feel that the Wikipedia page lacks in explaining the methods to tackle this and the paradigm I mention below).

I recently attended a lecture by Emmanuel Candès, where he introduced the Big-Data paradigm as

Collect data first $\Rightarrow$ Ask questions later

This is the main difference from hypothesis-driven research, where you first formulate a hypothesis and then collect data to say something about it.

He went a lot into the issues of quantifying reliability of hypotheses generated by data snooping. The main thing I took out of his lecture was that we really need to start to control the FDR and he presented the knockoff method to do so.

I think that CV should have a question on what is Big-Data and what is your definition on it. I feel that there are so many different "definitions", that it is hard to really grasp what it is, or explain it to others, if there is not a general consensus on what it consists of.

I feel that the "definition/paradigm/description" provided by Candès is the closest thing I agree on, what are your thoughts?

EDIT2: I feel that the answer should provide something more than just an explanation of the data itself. It should be a combination of data/methods/paradigm.

EDIT3: I feel that this interview with Michael Jordan could add something to the table as well.

EDIT4: I decided to choose the highest voted answer as the correct one. Although I think that all the answers add something to the discussion and I personally feel that this is more a question of a paradigm of how we generate hypotheses and work with data. I hope this question will serve as a pool of references for those that go looking for what Big-Data is. I hope that the Wikipedia page will be changed to further emphasize the multiple comparison problem and control of FDR.

• "Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it." Simon Matthews – Alexander Lutsenko Sep 18 '15 at 12:14
• this quote is not valid anymore. People are making many extraordinary works recently. If you look at the competitions on Kaggle, companies are improving their business and are earning alot of money by spending not alot of money. Other examples for the applications of Big Data can be found here: linkedin.com/pulse/… – Metariat Sep 18 '15 at 13:02
• @XuanQuangDO, I agree. Don't take this quote seriously. – Alexander Lutsenko Sep 18 '15 at 13:08
• @XuanQuangDO: well, I'm sure some teenagers are having extraordinary sex, but it doesn't change the fact there's a lot of incompetent or misguided fumbling around as well, which people will ruthlessly mock ;-) – Steve Jessop Sep 18 '15 at 22:54
• Big Data is any thing which is crash Excel. - DevOps Borat – user39853 Sep 19 '15 at 15:16

I had the pleasure of attending a lecture given by Dr. Hadley Wickham, of RStudio fame. He defined it such that

• Big Data: Can't fit in memory on one computer: > 1 TB
• Medium Data: Fits in memory on a server: 10 GB - 1 TB
• Small Data: Fits in memory on a laptop: < 10 GB

Hadley also believes that most data can at least be reduced to managable problems, and that a very small amount is actually true big data. He denotes this as the "Big Data Mirage".

• 90% Can be reduced to a small/ medium data problem with subsetting/sampling/summarising
• 9% Can be reduced to a very large number of small data problems
• 1% Is irreducibly big

Slides can be found here.

• @GuðmundurEinarsson, I have just edited it in, thanks for the suggestion. – Chris C Sep 18 '15 at 12:41
• While I do not think clear cut bounds exist, I think this post is very insightful. When I was working in a SW company I interacted with a quite a few customers looking for "big-data solutions"; in reality they missed a 16 GB SODIMM. – usεr11852 says Reinstate Monic Sep 18 '15 at 20:50
• With 1TB SSDs nowadays, nonvolatile storage isn't too far from volatile storage in speed. I feel like I'd expect big data to be bigger than 1TB, maybe at least 50 TB or something. – user541686 Sep 19 '15 at 19:18
• With all respect to you and Hadley, bid data is not just about the volume. Usually bid data is defined via 3V and, more recently, 4V model (proposed by Gartner) - see answer by Dawny33 below. However, some experts (including from Gartner) consider yet another, and they argue the most important, V dimension, where V stands for business value. For example, refer to this post and this post. – Aleksandr Blekh Sep 20 '15 at 10:28
• @AleksandrBlekh Your comment contains both a nuanced discussion of controversy surrounding the criteria for "Big Data" among experts, and some references to support your claims about it. I think that you should consider converting it into an answer. – Silverfish Sep 20 '15 at 19:59

A data set/stream is called Big Data, if it satisfies all the four V's

• Volume
• Velocity
• Veracity
• Variety

Unless and until it isn't satisfied, the data set can't be termed as Big Data.

A similar answer of mine, for reference.

Having said that, as a data scientist; I find the Map-Reduce framework really nice. Splitting your data, mapping it and then the results of the mapper step are reduced into a single result. I find this framework really fascinating, and how it has benefitted the world of data.

And these are some ways how I deal with the data problem during my work everyday:

1. Columnar Databases: These are a boon for data scientists. I use Aws Red Shift as my columnar data store. It helps in executing complex SQL queries and joins less of a pain. I find it really good, especially when my growth team asks some really complex questions, and I don't need to say "Yeah, ran a query; we'd get it in a day!"
2. Spark and the Map Reduce Framework: Reasons have been explained above.

And this is how a data experiment is carried out:

• The problem to be answered is identified
• The possible data sources are now listed out.
• Pipelines are designed for getting the data into Redshift from local databases. Yeah, Spark comes here. It really comes handy during the DB's --> S3 --> Redshift data movement.
• Then, the queries and SQL analyses are done on the data in Redshift.

Yes, there are Big Data algorithms like hyper loglog, etc; but I haven't found the need to use them.

So, Yes. The data is collected first before generating the hypothesis.

• I aggree with these things, but I think the term Big Data covers something more than the data itself. It is also the methods applied to it and the paradigm of first collecting the data before generating hypotheses about it. – Gumeo Sep 18 '15 at 12:30
• @GuðmundurEinarsson I was in a hurry, so wanted to give you the best answer in a short time. So, I have now edited and expanded it with the workflow and understandings from my daily experiences with large data in the industry. – Dawny33 Sep 18 '15 at 17:06
• The four Vs are being inverted here as defining big data rather than being important notable properties of big data. Lots of examples can be made of big data without several of those 4 and some are even listed in the IBM infographic. – John Oct 19 '15 at 9:58
• @John Yes, the V's are indeed changing a lot. There is also an argument for a new V (Value) – Dawny33 Oct 19 '15 at 10:02
• I'm not saying they're changing, I'm saying you're reversing a description of some properties into a definition incorrectly. It's like someone describing important things to them about a dog as loyalty, laughter, and licking and someone else coming along and saying that's the definition of a dog. That said, I think you were more on the right track considering reversing the analysis direction but it just needs to be attached to the size of the data in a way. I think there are lots of good ways to do that and it would be great if you developed one. – John Oct 19 '15 at 10:10

I think the only useful definition of big data is data which catalogs all information about a particular phenomenon. What I mean by that is that rather than sampling from some population of interest and collecting some measurements on those units, big data collects measurements on the whole population of interest. Suppose you're interested in Amazon.com customers. It's perfectly feasible for Amazon.com to collect information about all of their customers' purchases, rather than only tracking some users or only tracking some transactions.

To my mind, definitions that hinge on the memory size of the data itself to be of somewhat limited utility. By that metric, given a large enough computer, no data is actually big data. At the extreme of an infinitely large computer, this argument might seem reductive, but consider the case of comparing my consumer-grade laptop to Google's servers. Clearly I'd have enormous logistical problems attempting to sift through a terabyte of data, but Google has the resources to mange that task quite handily. More importantly, the size of your computer is not an intrinsic property of the data, so defining the data purely in reference to whatever technology you have at hand is kind of like measuring distance in terms of the length of your arms.

This argument isn't just a formalism. The need for complicated parallelization schemes and distributed computing platforms disappears once you have sufficient computing power. So if we accept the definition that Big Data is too big to fit into RAM (or crashes Excel, or whatever), then after we upgrade our machines, Big Data ceases to exist. This seems silly.

But let's look at some data about big data, and I'll call this "Big Metadata." This blog post observes an important trend: available RAM is increasing more rapidly than data sizes, and provocatively claims that "Big RAM is eating Big Data" -- that is, with sufficient infrastructure, you no longer have a big data problem, you just have data, and you return back to the domain of conventional analysis methods.

Moreover, different representation methods will have different sizes, so it's not precisely clear what it means to have "big data" defined in reference to its size-in-memory. If your data is constructed in such a way that lots of redundant information is stored (that is, you choose an inefficient coding), you can easily cross the threshold of what your computer can readily handle. But why would you want a definition to have this property? To my mind, whether or not the data set is "big data" shouldn't hinge on whether or not you made efficient choices in research design.

From the standpoint of a practitioner, big data as I define it also carries with it computational requirements, but these requirements are application-specific. Thinking through database design (software, hardware, organization) for $10^4$ observations is very different than for $10^7$ observations, and that's perfectly fine. This also implies that big data, as I define it, may not need specialized technology beyond what we've developed in classical statistics: samples and confidence intervals are still perfectly useful and valid inferential tools when you need to extrapolate. Linear models may provide perfectly acceptable answers to some questions. But big data as I define it may require novel technology. Perhaps you need to classify new data in a situation where you have more predictors than training data, or where your predictors grow with your data size. These problems will require newer technology.

As an aside, I think this question is important because it implicitly touches on why definitions are important -- that is, for whom are you defining the topic. A discussion of addition for first-graders doesn't start with set theory, it starts with reference to counting physical objects. It's been my experience that most of the usage of the term "big data" occurs in the popular press or in communications between people who are not specialists in statistics or machine learning (marketing materials soliciting professional analysis, for example), and it's used to express the idea that modern computing practices meant hat there is a wealth of available information that can be exploited. This is almost always in the context of the data revealing information about consumers that is, perhaps if not private, not immediately obvious. The anecdote about a retail chain sending direct mailings to people it assessed were expectant mothers on the basis of their recent purchases is the classic example of this.

So the connotation and analysis surrounding the common usage of "big data" also carries with it the idea that data can reveal obscure, hidden or even private details of a person's life, provided the application of a sufficient inferential method. When the media report on big data, this deterioration of anonymity is usually what they're driving at -- defining what "big data" is seems somewhat misguided in this light, because the popular press and nonspecialists have no concern for the merits of random forests and support vector machines and so on, nor do they have a sense of the challenges of data analysis at different scales. And this is fine. The concern from their perspective is centered on the social , political and legal consequences of the information age. A precise definition for the media or nonspecialists is not really useful because their understanding is not precise either. (Don't think me smug -- I'm simply observing that not everyone can be an expert in everything.)

• This. "It's been my experience that most of the usage of the term 'big data' occurs in the popular press or in communications between people who are not specialists in statistics or machine learning (marketing materials soliciting professional analysis, for example)" – Momo Sep 18 '15 at 14:43
• I think you hit the nail on the head with your last paragraph. I think that there is a very clear gap between the popular press understanding and what people in statistics/ML/Data science think of the term big data. I just feel that there needs to be a more clear consensus on what it actually is. One reason for that is to have a reference such that people cannot manipulate the term or misuse when it is obviously not applicable. – Gumeo Sep 18 '15 at 15:22
• I think I am starting to agree with you more and more. I still feel that CV needs a reference question, where people that are interested and into the subject put in their two cents on the matter. I was looking for questions here and I felt that this discussion was lacking. – Gumeo Sep 18 '15 at 19:33
• I think that this is a valuable conversation to have, and I'm glad you asked the question! And I'm glad you've found my comments helpful. – Sycorax says Reinstate Monica Sep 18 '15 at 21:01
• I love this answer for multiple reasons. First, I think it is really important to emphasize that "big data" has little to do with the algorithms used to analyze it. Most of them are 20-30 years old (random forest, linear regression, so on) and they work just fine. Some people in industry believe that "big data" comes together with fancy new algorithms, because they probably didn't even know that machine learning has existed for many years. Second, "big data" is not about size. If you have a server available with 128 Gb RAM and can fit everything into memory that is just great. (cont.) – skd Sep 25 '15 at 16:14

Crosschecking the huge literature on Big Data, I have collected up to 14 "V" terms, 13 of them along about 11 dimensions:

• Validity,
• Value,
• Variability/Variance,
• Variety,
• Velocity,
• Veracity/Veraciousness,
• Viability,
• Virtuality,
• Visualization,
• Volatility,
• Volume.

The 14th term is Vacuity. According to recent a provocative post, Big Data Doesn’t Exist. Its main points are that:

• “Big Data” Isn’t Big
• Most “Big Data” Isn’t Actually Useful
• [We should be] Making The Most Of Small Data

A proper definition of Big Data would evolve with hardware, software, needs and knowledge, and probably should not depend on a fixed size. Hence, the seizable defintion in Big data: The next frontier for innovation, competition, and productivity, June 2011:

"Big data" refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.

• The "vacuity" referenced article seems awfully weak. There's an implication that 30GB of data / day isn't big (and that size is the sole component of the definition). Further, it's argued that because companies say their data is much bigger than it actually is means it isn't big. There's no definition of big given anywhere. And all of the examples used to suggest "not big" have many of the V's listed here. – John Oct 19 '15 at 11:49
• The "vacuity" does not apply to the size only. Indeed, in the final one-liner, the definition of big is meant to evolve with the current state of practice. What was big in the past can be considered small a few years later. Here, I was using the term for the case where "big data" is used as a magic mantra with few substance, as illustrated in the above cartoon. – Laurent Duval Oct 19 '15 at 12:02
• How likely it is that 14 criteria will all start with a letter V? We're all stats minded people here, c'mon! – Aksakal Jan 20 '17 at 16:28
• Basically, I agree, this was only to show that a term like Big data is more likely to belong to the realm of marketing than to statistics. However, I wanted to share my "collection" of the terms I have read. It has started with 3V, then 5V, and sometimes 7 etc. Those term can vaguely help spot traits on the data one has – Laurent Duval Jan 20 '17 at 16:55

People seem to fixate on a big qualifier in Big Data. However, the size is only one of the components of this term (domain). It's not enough that your data set was big to call your problem (domain) a big data, you also need it be difficult to understand and analyze and even process. Some call this feature unstructured, but it's not just the structure it's also unclear relationship between different pieces and elements of data.

Consider the data sets that high energy physicists are working in places such as CERN. They've been working with petabytes size data for years before the Big Data term was coined. Yet even now they don't call this big data as far as I know. Why? Because the data is rather regular, they know what to do with it. They may not be able to explain every observation yet, so they work on new models etc.

Now we call Big Data the problems that deal with data sets that have sizes that could be generated in a few seconds from LHC in CERN. The reason is that these data sets are usually of data elements coming from multitude of sources with different formats, unclear relationships between the data and uncertain value to the business. It could be just 1TB but it's so difficult to process all the audio, vidio, texts, speech etc. So, in terms of complexity and resources required this trumps the petabytes of CERN's data. We don't even know if there's discernible useful information in our data sets.

Hence, Big Data problem solving involves parsing, extracting data elements of unknown value, then linking them to each other. "Parsing" an image can be a big a problem on its own. Say, you're looking for CCTV footage from the streets of the city trying to see whether people are getting angrier and whether it impacts the road accidents involving pedestrians. There's a ton of video, you find the faces, try to gauge their moods by expressions, then link this to the number of accidents data sets, police reports etc., all while controlling for weather (precitipotation, temperature) and traffic congestions... You need the storage and analytical tools that support these large data sets of different kinds, and can efficiently link the data to each other.

Big Data is a complex analysis problem where the complexity stems from both the sheer size and the complexity of structure and information encoding in it.

• Good input. The contrast you give between the LHC and the CCTV data problem is something that people often miss. – Gumeo Jan 20 '17 at 10:16

I think the reason why people get confused of what is Big Data is that they doesn't see its benefits. The value of Big Data (technique) is not only on the amount of data that you can collect, but also on the Predictive Modeling, which is eventually more important:

1. Predictive Modeling changed completely the way we do statistics and predictions, it gives us greater insight on our data, because new models, new techniques can detect better the trends, the noises of the data, can capture "multi"-dimensional database. The more dimentions we have in our database, the better chance we can creat the good model. Predictive Modeling is the heart of Big Data's value.
2. Big Data (in term of data size) is the preliminary step, and is there for serving the Predictive Modeling by: enrich the database with respect to the: 1.number of predictors (more variables), 2.number of observations.

More predictors because we are now able to capture the data that were impossible to capture before (because of the limited hardware power, limited capacity to work on the unstructured data). More predictors mean more chances to have the significant predictors, i.e better model, better prediction, better decision can be made for the business.

More observations not only make the model more robust over the time, but also help the model learn/detect every possible patterns that can be presented/generated in the reality.

The tricky thing about Big Data vs. its antonym (presumably Small Data?) is that it is a continuum. The big data people have gone to one side of the spectrum, the small data people have gone to the other, but there's no clear line in the sand that everyone can agree upon.

I would look at behavioral differences between the two. In small data situations, you have a "small" dataset, and you seek you squeeze as much information as possible our of every data-point you can. Get more data, you can get more results. However, getting more data can be expensive. The data one collects is often constrained to fit mathematical models, such as doing a partial factorial of tests to screen for interesting behaviors.

In big data situations, you have a "big" dataset, but your dataset tends not to be as constrained. You usually don't get to convince your customers to buy a latin-square of furniture, just to make the analysis easier. Instead you tend to have gobs and gobs of poorly structured data. To solve these problems, the goal tends not to be "select the best data, and squeeze everything you can out of it," like one might naively attempt if one is used to small data. The goal tends to be more along the lines of "if you can just get a tiny smidgen out of every single datapoint, the sum will be huge and profound."

Between them lies the medium sized data sets, with okay structure. These are the "really hard problems," so right now we tend to organize into two camps: one with small data squeezing every last bit out of it, and the other with big data trying to manage to let each data point shine in its own right. As we move forward, I expect to see more small-data processes trying to adapt to larger data-sets, and more big-data processes trying to adapt to leverage more structured data.

• Your characterization of small data sounds a lot like Bem's book on analysis. Please look up critiques of that because it's an inappropriate way to treat small datasets other than for private exploration on which to base future data collection. – John Oct 19 '15 at 11:37
• @John I may have to look at those. Are the critiques sufficient that I cannot even use the characterization as a point to describe a continuum? – Cort Ammon - Reinstate Monica Oct 19 '15 at 18:08
• It's really to long to go into here but the essential message is that you do not squeeze everything you can out of each datapoint with small data. Perhaps google Gelman and Forking Paths; or, experimenter degrees of freedom. You have to think about small and big data analysis differently, not just as points on a continuum. – John Oct 19 '15 at 19:07

I'd say there are three components that are essential in defining big data: the direction of analysis, the size of the data with respect to the population, and the size of the data with respect to computational problems.

The question itself posits that hypotheses are developed after data exists. I don't use "collected" because think the word "collected" implies for a purpose and data often exists for no known purpose at the time. The collecting often occurs in big data by bringing existing data together in service of a question.

A second important part is that it's not just any data for which post hoc analysis, what one would call exploratory analysis with smaller datasets, is appropriate. It needs to be of sufficient size that it's believed that estimates gathered from it are close enough to population estimates that many smaller sample issues can be ignored. Because of this I'm a little concerned that there is a push right now in the field toward multiple comparison corrections. If you had the whole population, or an approximation that you have good reason to believe is valid, such corrections should be moot. While I realize that it does occur that sometimes problems are posed that really do turn the "big data" into a small sample (e.g. large logistic regressions), that comes down to understanding what a large sample is for a specific question. Many of the multiple comparison questions should instead be turned to a effect size questions. And, of course, the whole idea you'd use tests with alpha = 0.05, as many still do with big data, is just absurd.

And finally, small populations don't qualify. In some cases there is a small population and one can collect all of the data required to examine it very easily and allow the first two criteria to be met. The data needs to be of sufficient magnitude that it becomes a computational problem. As such, in some ways we must concede that "big data" may be a transient buzz word and perhaps a phenomenon perpetually in search of strict definition. Some of the things that make "big data" big now will vanish in a few short years and definitions like Hadley's, based on computer capacity, will seem quaint. But at another level computational problems are questions that aren't about computer capacity or perhaps about computer capacity that can never be addressed. I think that in that sense the problems of defining "big data" will continue in the future.

One might note that I haven't provided examples or firm definitions of what a hard computational problem is for this domain (there are loads of examples generally in comp sci, and some applicable, that I won't go into). I don't want to make any because I think that will have to remain somewhat open. Over time the collected works of many people come together to make such things easy, more often through software development than hardware at this point. Perhaps the field will have to mature more fully in order to make this last requirement more solidly bounded but the edges will always be fuzzy.

• thanks for your input! I think that you provide valuable insights for this thread. I think that the size of the data w.r.t. the population has been somewhat overlooked here. – Gumeo Oct 19 '15 at 10:52

Wikipedia provides quite clear definition

Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate. (source https://en.wikipedia.org/wiki/Big_data)

other simple definition I know is

Data that does not fit computer memory.

Unfortunately I do not remember reference for it. Everything else emerges from this definitions - you have to deal somehow with big amounts of data.

I would add that Big Data is a reference to either working on big data-set (millions and/or billions of rows) or trying to find information/patterns on broad data resources you can collect now everywhere.