What is the difference between sampling statistics and big data statistics?

Question

What does significance mean in a classical sampling setting vs a big data setting?

Suppose there is a population P of events (P being arbitrarily large), and I randomly sample 50 of them. Suppose regression analysis reports a significant relationship between a dependent variable and a regressor, a p-value of 1%. I interpret that as saying there is a 1% chance that the points I sampled give me a false positive; a 1% chance that the points combine to look like a relationship when there really isn't one.

Suppose I have a "big data" set, and in it I have the data on every event. Maybe it's a business with meticulous records on transactions. Then if I analyze the whole data set, and I get a 1% p-value, I don't interpret that as giving insight on the sample as before, I interpret it as saying there's a 1% chance that randomness in the data generating process has produced such a relationship throughout all the data.

But if my data set is so large that I sample from it to generate statistics, I am back to thinking that a p-value says something mostly about the sampling and the probability of drawing such a relationship from an underlying distribution, as in the first case.

In other words I see two levels of randomness in data analysis. One is the effect of randomness / an error term in a data generating process, and one is which data points you happen to sample out of a population.

I have a few questions. Which aspect of randomness is the p-value supposed to describe, or is it both? Can classical regression analysis be done on "full" big data sets, or are certain assumptions violated? Does regression work equally well in any of these settings, just with different qualitative interpretations?

Any recommendations for literature on this?

Answer 1

Suppose I have a "big data" set, and in it I have the data on every event. Maybe it's a business with meticulous records on transactions. Then if I analyze the whole data set, and I get a 1% p-value, I don't interpret that as giving insight on the sample as before, I interpret it as saying there's a 1% chance that randomness in the data generating process has produced such a relationship throughout all the data.

First, let's talk about what is p-value linked to. Let's say we're interested in a parameter that describes our data. I'll pick the simplest one, the mean. So, we calculated a mean using our favorite formula for averages: $$\bar x=\frac 1 n \sum_{i=1}^nx_i$$

If our data is a random sample from the population, then we'd use this formula as an estimator of the sample mean $$\hat\mu=\bar x$$ Note, that this is a very good estimator usually, it has all these nice properties we want from the estimators, which I'm not going to go through here.

Now, notice that I market the estimator of the mean with a hat $\hat\mu$. The reason I'm doing this is because if there were another set of data that I could use, then the estimated value of the mean would have been different. So, the estimated value has uncertainty in it. We can use this value and the uncertainty to make statements about the population mean $\mu$, which we will never know, perhaps. So, we can make statements about how likely is the population mean $\mu$ (or true , in some sense, value) to be greater than ZERO, for instance. The statements are often accompanied by the p-values.

Summarizing, p-values are used when we only have the random sample from the population, but want to make statements about the unknown population parameters while we only have their sample estimates.

Now, if your data is the population, i.e. there is no other data set, the data that you have is all that is there and can exist, then you can calculate the population mean. There's no uncertainty anymore, you got $$\mu=\bar x$$ This is it, no need to guess or theorize about probabilities or likelihood, you know the population mean $\mu$. Anything you want to know about it is in its value. Hence, here we don't need p-values or hypotheses.

Note, that "big data" does not mean population. It can simply be the big sample. In this case may look up "large sample statistics" keywords. Usually, p-values are associated with small samples or finite samples, while "big data" may fall into a part of statistics that deals with asymptotic and infinite samples.

Also, small data set doesn't mean sample. For instance, if the question you asked was "what is the average grade in 7th grade in this particular school in Washington DC?", then you can go to school records and get grades of every student. You may get 100 observations comprising the full population, it's not a sample anymore. On the other hand, if your question was "what is the average grade of 7th graders in US?", then the same 100 records long data set is not a population anymore, but a sample.

So, the fact that your data set is a random sample or the population is not defined by the size of your data set, but by the question asked, by how the data was collected, what it represents etc.

Second, given what I just wrote, does your example actually represent the population? Likely, not. The fact that you brought up DGP (data generating process) usually means that this process generated a sample from the population that it could possibly generate. Otherwise, why would there be any uncertainty in it? If your data set contains every possible realization of the process, then you got the population and can calculate (as opposed to estimate) the parameters of the process and don't need p-values anymore. Again, with DGP the bigness of your data will likely lead to asymptotic estimators working very well.

Answer 2

Yes, classical regression analysis can be performed on big data sets - it's not having a large number of records that violates assumptions but other conditions such as nonlinearity, lack of independent errors, heteroscedasticity, and multicollinearity. Recall that a p-value estimates the probability of observing a given statistic, such as a z-score, having a magnitude the same or greater than the observed value given that the null hypothesis (and underlying statistical distribution) is actually true. p-values still retain the same meaning whether you're conducting hypothesis tests on thousands or billions of records.

Usually it's very small datasets, for example less than 10 observations per variable in a multiple regression analysis, where normality assumptions are be violated, and even then a Student's t-distribution score may be used as a better approximation of the standard error distribution.

With big datasets you may have the power to detect even slight associations between variables. For example, all of the coefficient estimates in a multiple regression may end up being statistically significant, with p-values well below the standard 0.05 threshold. In those situations, the effect sizes themselves become more useful for interpreting results rather than the p-values.

Keep in mind even when analyzing the whole data set with billions of records, in a sense you're still sampling from a much larger (theoretically infinite) population since you're drawing records from a limited number of observations during a finite period of time. Machine learning and data mining techniques are still making inferences for this larger population, even when there are no p-values or standard errors.