When sampling a population for surveys we can often limit our sample size to hundreds, but when doing a Monte Carlo simulation we need way more. Why?

Question

I’m a bit of a stats-noob, so I am not sure I will manage to formulate this question properly, but let me do my best.

I‘m trying to develop an intuition for sample sizes and when they are sufficient to get to a reasonable degree of representativeness of a larger population.

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations. (This is at least what calculators such as this one are telling me.)

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model? Every simulation can be thought of as representing a sample of the larger population of all possible simulations (the solution space).

The solution space of a multi-dimensional problem can be very large, but the survey/poll scenario tells me that this doesn’t mean that the sample size will need to be large.

This seems to suggest that any application of the Monte Carlo method can be concluded within several hundred/thousand of simulations as well. But I feel this cannot be true, as it would appear to defeat the purpose of the plethora of “more efficient alternatives” to Monte Carlo (such as Markov Chain Monte Carlo).

What am I missing? What is the difference between surveying a large population of people on their election vote (for example) and “surveying” a large population of parameter permutations on their performance (when plugged into a certain model)?

Why does a surprisingly small number of surveys suffice to get a decent idea of the likely outcome of an election, while a small number of Monte Carlo simulations does not suffice (I think) to get a decent idea of the likely (top) performance of a model?

My apologies for needing too many words, but it’s the best I can do I am afraid. I hope you catch my drift regardless and can steer me in the right direction. Thanks a lot!

Answer 1

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations.

In the calculator you linked to, it only considers estimating a single proportion. The relationship between sample size, confidence level, and desired margin of error is simple for estimating a single proportion from iid binomial data. Political polls, at least, tend to be focused on just a few proportions (what % of voters favor candidate A over candidate B). If they assume random sampling and ignorable nonresponse, they can use such a calculator to find that, say, ~1000 respondents will get you a 95% MOE of $\pm$ 3 percentage points. Even if you wanted to do multiple comparisons corrections (though they often don't) for reporting a handful of proportions at once, ~1000 respondents is typically still good enough for reasonably narrow MOEs.

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model?

If you're studying a higher-dimensional space, you need more data if you want to honestly account for the uncertainty in studying many estimates at once. If your estimators have some complicated intractable distribution, sample-size calculations may be sketchy and you want to err on the side of more data. If you want to characterize the space in more detail than just a single (multi-dimensional) point estimate, you need more data.

In fact... this is true of surveys as well. There are plenty of much larger surveys, such as the ones that are run by national statistical offices (such as the US Census Bureau). Some ask simple binary questions where the above sample-size calculator works, but they want bigger samples in order to account for asking many such questions. Other questions are quantitative measurements and might need a different approach to estimate the sample size. They also often need bigger sample sizes in order to get precise sub-group estimates (think small geographic regions or small demographic groups). So just a few hundred responses is not always enough. For instance, the American Community Survey collects roughly 3 million responses each year.

Answer 2

Surveys are relatively expensive, which determines a different balance between the cost and value. One instance of a Monte Carlo simulation is extremely cheap so you can repeat it more until the marginal cost of the ongoing simulation exceeds its marginal value.

Answer 3

Population size is misleading. Consider the following two experiments:

Experiment 1: A coin lands on heads with unknown probability $p$. We flip the coin $100$ times and try to estimate $p$.

Experiment 2: We sample uniformly from a bag containing a trillion balls, either red or blue. The probability of sampling a red ball is $p$. We sample $100$ balls and try to estimate $p$.

These experiments might seem very different, but probabilistically they are extremely similar. Abstracting away the population size, we are trying to compute a single parameter value, $p$, and the value of $p$ has such a big effect on our observations that even with only $100$ samples the value of $p$ can be well-estimated. With more complicated models, there may be many parameters whose impact (individually and collectively) on our observations is very subtle, and so many samples are needed to tease out these subtleties.

This line of thinking leads one into the intersection of probability theory and information theory. Essentially, a few observations of flipping a coin carry a lot of information about the parameter $p$, but for more complicated models the amount of information in each individual observation may be very small. It is this that governs how many samples you need, and not the total size of the available data.

Answer 4

One important side note:

This seems to suggest that any application of the Monte Carlo method can be concluded within several hundred/thousand of simulations as well. But I feel this cannot be true, as it would appear to defeat the purpose of the plethora of “more efficient alternatives” to Monte Carlo (such as Markov Chain Monte Carlo).

Markov Chain Monte Carlo is almost always not more efficient than a Monte Carlo approach with independent samples, but rather much, much less efficient. With most MCMC algorithms, each sample is positively correlated with the previous sample. The impact of this correlation can be quite extreme: it's not too uncommon to see every one thousand samples being equal to one new sample (i.e. takes one thousand samples to forget the previous state in the MCMC algorithm).

In these scenarios, if we you were content with the inference learned from 100 independent samples, you would need 100,000 samples from an MCMC algorithm with the heavy correlation mentioned above.

MCMC is generally used when you cannot take independent draws from the target distribution.

Answer 5

I have doubts that the statement "a surprisingly small number of surveys suffice to get a decent idea of the likely outcome of an election" actually holds well in practice. Well, maybe depends on your definition of "decent"...

Caveat: What I know about polling is mostly from reading FiveThirtyEight. If you want to learn more about the accuracy of polls, in the US at least, take a look at some FiveThirtyEight posts tagged "polling accuracy".

To start with, you are considering a very simple problem: estimate a proportion by sampling a homogeneous population. Voting intent on the other hand depends on a number of factors (personal characteristics, the state of the economy, other current questions/concerns relevant to voters). So polling companies don't survey respondents (completely) at random: they want to have samples that are representative of the population (in a county, a state or a country). So surveys are designed by using census data to determine how to construct a representative sample efficiently. The sample itself can be biased (it may not be efficient to over-sample the most common sub-group of voters) but the bias can be corrected to calculate an unbiased estimate of voting preferences.

Another point to consider is that when an election is close, it's probably necessary to have a smaller margin of error and therefore collect a larger sample of respondents.

The second part of your question about Markov chain Monte Carlo methods has a similar issue of making a simplistic generalization. No one wants to waste their time and computational resources on running their MCMC sampling for longer than necessary to get convergence. That's why it's so important to have tools to diagnose convergence issues. Some chains may need 1,000 samples; others 1,000,000.