# Does sampling from a large dataset lead to correct inferences?

Say we have some population, and we obtain a "representative" random sample of that population, $$(y_i, x_i)_{i = 1}^n$$, where $$n$$ is very large (millions) and $$x_i = (x_{i1}, x_{i2}, ... x_{ip})'$$ is a multivariate predictor of the response $$y_i$$.

This scenario is fairly common when working with large, modern data sets, but still, suppose we wanted to conduct some meaningful inferences about the data using (say) regression.

Assuming we had limited computational power, one approach is to draw a random sample from the larger data. That is, we can draw some sample $$(y_j^*, x_j^*)_{j =1}^k$$ from $$(y_i, x_i)_{i = 1}^n$$, where $$k << n$$.

Assuming this subsample is representative of the larger sample, can we use the subsample to make inferences about the larger, original population?

My thinking is that, yes, this works, our regression coefficients $$\beta$$ derived from the subsample should reflect the coefficients of the larger sample, which then allows us to make inferences about the population, albeit with slightly higher variance. But if the subsample is large enough (say a million?) then this variance shouldn't be that problematic, since most forms of regression are consistent and we are dealing with a large number of data points. Thoughts?

Edit: On my use of the word "representative", I found this thread:

What exactly does 'representative sample' refer to?

Perhaps as a working definition, we could say that the sample is drawn without bias from the population. Or perhaps: if we could fit the regression on the entire data, we would get unbiased estimates of $$\beta$$. If the sample were not "representative" then the estimates naturally have some bias, no?

Second question: how would the possible answer to this question change depending on what we mean by representative?

On the word "Random": If our extremely large data set consisted of data points that were obtained via a random sample, then I see no issue with subsampling, but of course, I would like to hear others' thoughts on this. What if the sampling mechanism were not entirely known? Can we still take a simple random subsample?

More generally, under what conditions would a subsample lead to correct inferences about a population?

• Could you explain what you mean by "representative"? This term is frequently used, but it has been argued that it is poorly defined and has inconsistent meanings among different communities of analysts. Moreover, the correct answer to this question could dramatically change depending on what you mean. The existing answers assume you mean "random" in some sense, but that's rarely the case for very large datasets. – whuber Jan 15 '19 at 21:30
• Thank you for the characterization of "representative." It raises serious questions, though, such as what "bias" could possibly mean unless some form of randomization is involved. If by "bias" you mean the difference between the estimated coefficients and the actual ones, then you have ruled out the possibility of even using sampling to make estimates due to the circularity of this definition! After all, how can you ensure lack of bias (in this second sense) without already knowing the values you are trying to estimate? – whuber Jan 16 '19 at 13:23
• I can't tell what you're trying to do with your comment. Statisticians frequently discuss "bias" and the presence or non-presence of it in a variety of methods. E.g., we frequently care about the consistency properties of estimators, even though "in the real world" we never can be sure how close our estimate is to the truth. It seems to me that you're intentionally being uncharitable with your interpretation of my question. Can we use a subsample from a larger sample to make inferences about a population? That is all that I am asking. – Marcel Jan 16 '19 at 16:53
• You cannot even discuss bias or consistency unless the sample is random. I'm not being uncharitable--I'm focusing on the chief distinction between "big data" and truly representative samples because it is crucial for the correct interpretation of your question and any answers it might get. If the original sample hasn't the properties needed to perform statistical inference, then no amount of subsampling will improve it. – whuber Jan 16 '19 at 17:29
• Your issue is the lack of the word random? You seem to be honing in on the phrase "Big Data" as if it is mutually exclusive from the word "Random". If that is the issue, then I will update my post. – Marcel Jan 16 '19 at 17:33

If you have the whole population, you are not really doing any inference of a variable, that only happens when you are taking a sample. Let's say you are using a model that predicts weight based on height, so it's

$$w = a\cdot h + \epsilon$$

Where $$\epsilon$$ is some error. Somehow you have collected data on the whole population of the planet. Or even better, any person that ever lived. But then, what's a person? Already, you have some interesting questions here.

But let's stick to the plan. For the whole population, you estimate an 'a'. Then we sample down to 1 million people. If it's a random sample, you can establish limits on how far your inference on this sample lies from the true value. In frequentist statistics, you assume a 'true' value of $$a$$, which is going to be superclose to the inference on the whole population, and its going to be very close to the inference on the smaller population as well. Under assumptions on the error, the variance of an estimator will be proportional to 1 over the square of 1 million for the sample, and 1 over the square of whatever the size of the whole population for that original inference. This follows from the Central Limit Theorem. So both are close to the 'true' value and they are going to be close to each other as well.

I mentioned frequentist, so now I have to mention some other viewpoint as well, but the inference in Bayesian statistics is going to be pretty much equal as well, allthough perhaps you are not really assuming a 'true' value for $$a$$, but rather updating your belief after measuring all those people. But the math still holds pretty much and if you do it with the sample it will be extremely close to the inference on the whole population.

Regardless of the estimator variance, a more interesting point here is that the model is clearly not the truth. There is no true value for $$a$$, it's just a simplification that you may trust to use for your usecase. This holds for any model, however sophisticated it may appear.

Another thought, if you have big data it is often the case that you have a lot of data relative to the number of variables that you are estimating. At that scale, using the Central Limit Theorem for deriving estimator variances is sometimes missing the larger point, as in the previous paragraph, your model is wrong and you already know it. For example if you use a simple linear regression such as above, with a population of 1m people, your estimator variance is in the order of 1 over square root of 1 million, that's 0.001. So your report is going to be, "$$a$$ is contained in the interval $$[1.234, 1.235]$$. The significance is through the roof.". But at that point, a better question may be, how well does this model actually predict weight from height? And you apply cross-validation and things like that, and it's going to look like machine learning more.

• This is a very nice discussion on the issue of very large samples, but somehow it is not quite answering the question. In my question, we are supposing that there exists a population, and then we are taking an extremely large sample, which we then subsample to a more reasonable size so that we can actually fit a model. The population is the interest, but the sample is too large to work with, so we have to make a trade and use a subset. Of course, when you have millions of data points, your variance shrinks to nothing by anything that's a consistent estimator – Marcel Jan 16 '19 at 0:53
• When we are in a big data environment, where we still want to conduct some inference, how do we go about doing it? Everything will flag as significant, no matter how small the effect size, so we can't use that as a metric. But we may be still interested in the effect sizes of certain covariates on an outcome. How do we do that? Are you suggesting model selection based on predictive accuracy? I thought that introduced bias, generally speaking – Marcel Jan 16 '19 at 0:56
• Model selection using statistical significance introduces bias as well. To avoid bias, pre-specify the model or use some form of cross validation. – Great38 Jan 16 '19 at 3:42

Yes, this works. All data is a sample population. If you have enough to achieve some level of performance on some metric, than you have achieved your goal. There will generally be a point of diminishing returns on the size of the data. Thus, more data will make little difference. As long as you have enough to make an appropriate generalization on test data, then you are good. Additionally, you can used unsampled instances from the larger datset for testing.

I imagine it depends on what kind of inferences you're trying to make about the population. In general you could be making any kind of inference, including inferences about the estimators used to learn population parameters. This question immediately made me think of the bootstrap and jackknife resampling techniques, which are often used to make inferences about the variance of estimators.

These methods fail in at least some circumstances: the jackknife fails to estimate the variance of the sample median. So although it probably works in some circumstances, it does not work for all subsampling techniques, over all classes of inferences.

This is certainly the case in the infinite population setting, but IRL that's rarely the case at least as far as big data are concerned.

For instance, if I run an insurance company and manage claims, I can do statistical analyses of all claims or even just a large subset. There's a challenge here. If I do a simple random sample (SRS) of n/N > 0.3 (30% or more) of my claims, then the normal approach to calculating CIs and p-values will not replicate. Because if the study were done again, I would be very likely to sample exactly one of the same n's I pulled in the first iteration, and in the second, and so on, meaning that my estimate of the SE will be too large (if the data are truly independent) or possibly too small (if there is dependence).

Finite sample corrections can be used under the independent data assumption. Correctly identifying correlation structures is a requirement to estimate the correcting sampling distribution of statistics.

• What does IRL stand for? – Richard Hardy Jan 24 '19 at 10:14
• @RichardHardy in real life – AdamO Jan 24 '19 at 13:49