Does sampling from a large dataset lead to correct inferences?

Question

Say we have some population, and we obtain a "representative" random sample of that population, $(y_i, x_i)_{i = 1}^n$, where $n$ is very large (millions) and $x_i = (x_{i1}, x_{i2}, ... x_{ip})'$ is a multivariate predictor of the response $y_i$.

This scenario is fairly common when working with large, modern data sets, but still, suppose we wanted to conduct some meaningful inferences about the data using (say) regression.

Assuming we had limited computational power, one approach is to draw a random sample from the larger data. That is, we can draw some sample $(y_j^*, x_j^*)_{j =1}^k$ from $(y_i, x_i)_{i = 1}^n$, where $k << n$.

Assuming this subsample is representative of the larger sample, can we use the subsample to make inferences about the larger, original population?

My thinking is that, yes, this works, our regression coefficients $\beta$ derived from the subsample should reflect the coefficients of the larger sample, which then allows us to make inferences about the population, albeit with slightly higher variance. But if the subsample is large enough (say a million?) then this variance shouldn't be that problematic, since most forms of regression are consistent and we are dealing with a large number of data points. Thoughts?

Edit: On my use of the word "representative", I found this thread:

What exactly does 'representative sample' refer to?

Perhaps as a working definition, we could say that the sample is drawn without bias from the population. Or perhaps: if we could fit the regression on the entire data, we would get unbiased estimates of $\beta$. If the sample were not "representative" then the estimates naturally have some bias, no?

Second question: how would the possible answer to this question change depending on what we mean by representative?

On the word "Random": If our extremely large data set consisted of data points that were obtained via a random sample, then I see no issue with subsampling, but of course, I would like to hear others' thoughts on this. What if the sampling mechanism were not entirely known? Can we still take a simple random subsample?

More generally, under what conditions would a subsample lead to correct inferences about a population?

Answer 1

If you have the whole population, you are not really doing any inference of a variable, that only happens when you are taking a sample. Let's say you are using a model that predicts weight based on height, so it's

$$w = a\cdot h + \epsilon$$

Where $\epsilon$ is some error. Somehow you have collected data on the whole population of the planet. Or even better, any person that ever lived. But then, what's a person? Already, you have some interesting questions here.

But let's stick to the plan. For the whole population, you estimate an 'a'. Then we sample down to 1 million people. If it's a random sample, you can establish limits on how far your inference on this sample lies from the true value. In frequentist statistics, you assume a 'true' value of $a$, which is going to be superclose to the inference on the whole population, and its going to be very close to the inference on the smaller population as well. Under assumptions on the error, the variance of an estimator will be proportional to 1 over the square of 1 million for the sample, and 1 over the square of whatever the size of the whole population for that original inference. This follows from the Central Limit Theorem. So both are close to the 'true' value and they are going to be close to each other as well.

I mentioned frequentist, so now I have to mention some other viewpoint as well, but the inference in Bayesian statistics is going to be pretty much equal as well, allthough perhaps you are not really assuming a 'true' value for $a$, but rather updating your belief after measuring all those people. But the math still holds pretty much and if you do it with the sample it will be extremely close to the inference on the whole population.

Regardless of the estimator variance, a more interesting point here is that the model is clearly not the truth. There is no true value for $a$, it's just a simplification that you may trust to use for your usecase. This holds for any model, however sophisticated it may appear.

Another thought, if you have big data it is often the case that you have a lot of data relative to the number of variables that you are estimating. At that scale, using the Central Limit Theorem for deriving estimator variances is sometimes missing the larger point, as in the previous paragraph, your model is wrong and you already know it. For example if you use a simple linear regression such as above, with a population of 1m people, your estimator variance is in the order of 1 over square root of 1 million, that's 0.001. So your report is going to be, "$a$ is contained in the interval $[1.234, 1.235]$. The significance is through the roof.". But at that point, a better question may be, how well does this model actually predict weight from height? And you apply cross-validation and things like that, and it's going to look like machine learning more.

Answer 2

Yes, this works. All data is a sample population. If you have enough to achieve some level of performance on some metric, than you have achieved your goal. There will generally be a point of diminishing returns on the size of the data. Thus, more data will make little difference. As long as you have enough to make an appropriate generalization on test data, then you are good. Additionally, you can used unsampled instances from the larger datset for testing.

Answer 3

I imagine it depends on what kind of inferences you're trying to make about the population. In general you could be making any kind of inference, including inferences about the estimators used to learn population parameters. This question immediately made me think of the bootstrap and jackknife resampling techniques, which are often used to make inferences about the variance of estimators.

These methods fail in at least some circumstances: the jackknife fails to estimate the variance of the sample median. So although it probably works in some circumstances, it does not work for all subsampling techniques, over all classes of inferences.

Answer 4

This is certainly the case in the infinite population setting, but IRL that's rarely the case at least as far as big data are concerned.

For instance, if I run an insurance company and manage claims, I can do statistical analyses of all claims or even just a large subset. There's a challenge here. If I do a simple random sample (SRS) of n/N > 0.3 (30% or more) of my claims, then the normal approach to calculating CIs and p-values will not replicate. Because if the study were done again, I would be very likely to sample exactly one of the same n's I pulled in the first iteration, and in the second, and so on, meaning that my estimate of the SE will be too large (if the data are truly independent) or possibly too small (if there is dependence).

Finite sample corrections can be used under the independent data assumption. Correctly identifying correlation structures is a requirement to estimate the correcting sampling distribution of statistics.