8
$\begingroup$

I’m a bit of a stats-noob, so I am not sure I will manage to formulate this question properly, but let me do my best.

I‘m trying to develop an intuition for sample sizes and when they are sufficient to get to a reasonable degree of representativeness of a larger population.

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations. (This is at least what calculators such as this one are telling me.)

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model? Every simulation can be thought of as representing a sample of the larger population of all possible simulations (the solution space).

The solution space of a multi-dimensional problem can be very large, but the survey/poll scenario tells me that this doesn’t mean that the sample size will need to be large.

This seems to suggest that any application of the Monte Carlo method can be concluded within several hundred/thousand of simulations as well. But I feel this cannot be true, as it would appear to defeat the purpose of the plethora of “more efficient alternatives” to Monte Carlo (such as Markov Chain Monte Carlo).

What am I missing? What is the difference between surveying a large population of people on their election vote (for example) and “surveying” a large population of parameter permutations on their performance (when plugged into a certain model)?

Why does a surprisingly small number of surveys suffice to get a decent idea of the likely outcome of an election, while a small number of Monte Carlo simulations does not suffice (I think) to get a decent idea of the likely (top) performance of a model?

My apologies for needing too many words, but it’s the best I can do I am afraid. I hope you catch my drift regardless and can steer me in the right direction. Thanks a lot!

$\endgroup$
3
  • $\begingroup$ What are the relative probabilities? Top performance often means a low probability of failure, and to get even one significant digit into the outcome of a simple Bernoulli trial you need to run the sim until you get 200 failure events or so. Personally I had to battle with this when working in telcomm, and the target quality of service was bit error probability in the ball park of $10^{-12}$ or so (I would run "quick Monte Carlo" sims pitting competing suggestions down to $10^{-9}$ and expecting the trend on the error curves as a function of SNR to continue). $\endgroup$ Oct 24, 2022 at 5:19
  • $\begingroup$ Can you re-phrase that? Samples of hundreds or thousands are often sufficient, for populations however large. Where is it written that you can, or might want to extend that to Monte Carlo simulations, of a multi-parameter model or not? How is any simulation thought of as representing a sample of the larger population of all possible simulations (the solution space) related to the Question or its exposition? The solution space of a multi-dimensional problem can be very large, and how does the survey/poll scenario tell you that doesn’t mean the sample size will need to be large. $\endgroup$ Oct 24, 2022 at 20:39
  • $\begingroup$ It is a horse of a different color. That is, if the objective of a procedure is to arrive at an answer that is precise to within a few percent, then that objective is not the same as testing whether A or B is more explanatory of C. The lesson is to first identify the objective of a test, and only thereafter worry about sample size to guarantee power. $\endgroup$
    – Carl
    Oct 25, 2022 at 9:53

5 Answers 5

12
$\begingroup$

I know that when doing surveys or polls, a sample size of mere hundreds or thousands is often sufficient, even for very large populations.

In the calculator you linked to, it only considers estimating a single proportion. The relationship between sample size, confidence level, and desired margin of error is simple for estimating a single proportion from iid binomial data. Political polls, at least, tend to be focused on just a few proportions (what % of voters favor candidate A over candidate B). If they assume random sampling and ignorable nonresponse, they can use such a calculator to find that, say, ~1000 respondents will get you a 95% MOE of $\pm$ 3 percentage points. Even if you wanted to do multiple comparisons corrections (though they often don't) for reporting a handful of proportions at once, ~1000 respondents is typically still good enough for reasonably narrow MOEs.

So now what happens if we extend this notion to Monte Carlo simulations of a multi-parameter model?

If you're studying a higher-dimensional space, you need more data if you want to honestly account for the uncertainty in studying many estimates at once. If your estimators have some complicated intractable distribution, sample-size calculations may be sketchy and you want to err on the side of more data. If you want to characterize the space in more detail than just a single (multi-dimensional) point estimate, you need more data.

In fact... this is true of surveys as well. There are plenty of much larger surveys, such as the ones that are run by national statistical offices (such as the US Census Bureau). Some ask simple binary questions where the above sample-size calculator works, but they want bigger samples in order to account for asking many such questions. Other questions are quantitative measurements and might need a different approach to estimate the sample size. They also often need bigger sample sizes in order to get precise sub-group estimates (think small geographic regions or small demographic groups). So just a few hundred responses is not always enough. For instance, the American Community Survey collects roughly 3 million responses each year.

$\endgroup$
16
$\begingroup$

Surveys are relatively expensive, which determines a different balance between the cost and value. One instance of a Monte Carlo simulation is extremely cheap so you can repeat it more until the marginal cost of the ongoing simulation exceeds its marginal value.

$\endgroup$
2
  • $\begingroup$ Wouldn’t the ”shape” of the solution space matter as well? I mean; a survey that comprises a single question that has a finite set of possible answers versus a complex model that takes multiple parameters as input — isn’t that relevant for the number of samples that are needed to get an representative sense of the distribution of the answers/solutions? $\endgroup$ Oct 22, 2022 at 18:36
  • 1
    $\begingroup$ "Surveys are relatively expensive" I mean, compared to simulations ... they're about the cheapest form of experimental psychology. $\endgroup$ Oct 24, 2022 at 18:20
11
$\begingroup$

Population size is misleading. Consider the following two experiments:

Experiment 1: A coin lands on heads with unknown probability $p$. We flip the coin $100$ times and try to estimate $p$.

Experiment 2: We sample uniformly from a bag containing a trillion balls, either red or blue. The probability of sampling a red ball is $p$. We sample $100$ balls and try to estimate $p$.

These experiments might seem very different, but probabilistically they are extremely similar. Abstracting away the population size, we are trying to compute a single parameter value, $p$, and the value of $p$ has such a big effect on our observations that even with only $100$ samples the value of $p$ can be well-estimated. With more complicated models, there may be many parameters whose impact (individually and collectively) on our observations is very subtle, and so many samples are needed to tease out these subtleties.

This line of thinking leads one into the intersection of probability theory and information theory. Essentially, a few observations of flipping a coin carry a lot of information about the parameter $p$, but for more complicated models the amount of information in each individual observation may be very small. It is this that governs how many samples you need, and not the total size of the available data.

$\endgroup$
2
  • 1
    $\begingroup$ Amazing answer! Super illuminating. Thank you very much. $\endgroup$ Oct 23, 2022 at 15:54
  • 1
    $\begingroup$ That said, if you are taking $n$ samples from a finite population of size $N$, the population size does play a role if $N$ is not very large. In the design-based inference framework used in survey sampling, variances can be multiplied by a "finite population correction" factor: $fpc = 1-(n/N)$. Say you take a simple random sample with replacement and find the sample mean; its variance is not $s^2/n$ but $(1-(n/N))s^2/n$. In other words, you can get the same margins of error with a smaller $n$ when $N$ is not large. $\endgroup$
    – civilstat
    Oct 23, 2022 at 18:24
7
$\begingroup$

One important side note:

This seems to suggest that any application of the Monte Carlo method can be concluded within several hundred/thousand of simulations as well. But I feel this cannot be true, as it would appear to defeat the purpose of the plethora of “more efficient alternatives” to Monte Carlo (such as Markov Chain Monte Carlo).

Markov Chain Monte Carlo is almost always not more efficient than a Monte Carlo approach with independent samples, but rather much, much less efficient. With most MCMC algorithms, each sample is positively correlated with the previous sample. The impact of this correlation can be quite extreme: it's not too uncommon to see every one thousand samples being equal to one new sample (i.e. takes one thousand samples to forget the previous state in the MCMC algorithm).

In these scenarios, if we you were content with the inference learned from 100 independent samples, you would need 100,000 samples from an MCMC algorithm with the heavy correlation mentioned above.

MCMC is generally used when you cannot take independent draws from the target distribution.

$\endgroup$
2
  • 2
    $\begingroup$ This answer makes an important point but is not completely accurate; if the autocorrelation is >0 (which it usually is) then the previous state is never truly "forgotten" after any finite number of steps. Instead, we tend to work with the effective sample size which for a simple 1D MCMC is (number of iterations) / (1 - autocorrelation). $\endgroup$
    – JDL
    Oct 24, 2022 at 8:48
  • 1
    $\begingroup$ +1 to both. The issue of "effective sample size" is very prominent went working with MCMC algorithms (and annoyingly it doesn't even have a universal answer (e.g. see stats.stackexchange.com/questions/429470) $\endgroup$
    – usεr11852
    Oct 25, 2022 at 0:45
0
$\begingroup$

I have doubts that the statement "a surprisingly small number of surveys suffice to get a decent idea of the likely outcome of an election" actually holds well in practice. Well, maybe depends on your definition of "decent"...

Caveat: What I know about polling is mostly from reading FiveThirtyEight. If you want to learn more about the accuracy of polls, in the US at least, take a look at some FiveThirtyEight posts tagged "polling accuracy".

To start with, you are considering a very simple problem: estimate a proportion by sampling a homogeneous population. Voting intent on the other hand depends on a number of factors (personal characteristics, the state of the economy, other current questions/concerns relevant to voters). So polling companies don't survey respondents (completely) at random: they want to have samples that are representative of the population (in a county, a state or a country). So surveys are designed by using census data to determine how to construct a representative sample efficiently. The sample itself can be biased (it may not be efficient to over-sample the most common sub-group of voters) but the bias can be corrected to calculate an unbiased estimate of voting preferences.

Another point to consider is that when an election is close, it's probably necessary to have a smaller margin of error and therefore collect a larger sample of respondents.

The second part of your question about Markov chain Monte Carlo methods has a similar issue of making a simplistic generalization. No one wants to waste their time and computational resources on running their MCMC sampling for longer than necessary to get convergence. That's why it's so important to have tools to diagnose convergence issues. Some chains may need 1,000 samples; others 1,000,000.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.