As sample size increases, why does the standard deviation of results get smaller? Can someone please provide a laymen example and explain why

Question

As sample size increases (for example, a trading strategy with an 80% edge), why does the standard deviation of results get smaller? Can someone please explain why standard deviation gets smaller and results get closer to the true mean... perhaps provide a simple, intuitive, laymen mathematical example.

Answer 1

As sample size increases (for example, a trading strategy with an 80% edge), why does the standard deviation of results get smaller?

The key concept here is "results." What are these results? The results are the variances of estimators of population parameters such as mean $\mu$.

For instance, if you're measuring the sample variance $s^2_j$ of values $x_{i_j}$ in your sample $j$, it doesn't get any smaller with larger sample size $n_j$: $$s^2_j=\frac 1 {n_j-1}\sum_{i_j} (x_{i_j}-\bar x_j)^2$$ where $\bar x_j=\frac 1 n_j\sum_{i_j}x_{i_j}$ is a sample mean.

However, the estimator of the variance $s^2_\mu$ of a sample mean $\bar x_j$ will decrease with the sample size: $$\frac 1 n_js^2_j$$

The layman explanation goes like this. Suppose the whole population size is $n$. If we looked at every value $x_{j=1\dots n}$, our sample mean would have been equal to the true mean: $\bar x_j=\mu$. In other words the uncertainty would be zero, and the variance of the estimator would be zero too: $s^2_j=0$

However, when you're only looking at the sample of size $n_j$. You calculate the sample mean estimator $\bar x_j$ with uncertainty $s^2_j>0$. So, somewhere between sample size $n_j$ and $n$ the uncertainty (variance) of the sample mean $\bar x_j$ decreased from non-zero to zero. That's the simplest explanation I can come up with.

Answer 2

Maybe the easiest way to think about it is with regards to the difference between a population and a sample. If I ask you what the mean of a variable is in your sample, you don't give me an estimate, do you? You just calculate it and tell me, because, by definition, you have all the data that comprises the sample and can therefore directly observe the statistic of interest. Correlation coefficients are no different in this sense: if I ask you what the correlation is between X and Y in your sample, and I clearly don't care about what it is outside the sample and in the larger population (real or metaphysical) from which it's drawn, then you just crunch the numbers and tell me, no probability theory involved.

Now, what if we do care about the correlation between these two variables outside the sample, i.e. in either some unobserved population or in the unobservable and in some sense constant causal dynamics of reality? (If we're conceiving of it as the latter then the population is a "superpopulation"; see for example https://www.jstor.org/stable/2529429.) Then of course we do significance tests and otherwise use what we know, in the sample, to estimate what we don't, in the population, including the population's standard deviation which starts to get to your question.

But first let's think about it from the other extreme, where we gather a sample that's so large then it simply becomes the population. Imagine census data if the research question is about the country's entire real population, or perhaps it's a general scientific theory and we have an infinite "sample": then, again, if I want to know how the world works, I leverage my omnipotence and just calculate, rather than merely estimate, my statistic of interest. What if I then have a brainfart and am no longer omnipotent, but am still close to it, so that I am missing one observation, and my sample is now one observation short of capturing the entire population? Now I need to make estimates again, with a range of values that it could take with varying probabilities - I can no longer pinpoint it - but the thing I'm estimating is still, in reality, a single number - a point on the number line, not a range - and I still have tons of data, so I can say with 95% confidence that the true statistic of interest lies somewhere within some very tiny range. It all depends of course on what the value(s) of that last observation happen to be, but it's just one observation, so it would need to be crazily out of the ordinary in order to change my statistic of interest much, which, of course, is unlikely and reflected in my narrow confidence interval.

The other side of this coin tells the same story: the mountain of data that I do have could, by sheer coincidence, be leading me to calculate sample statistics that are very different from what I would calculate if I could just augment that data with the observation(s) I'm missing, but the odds of having drawn such a misleading, biased sample purely by chance are really, really low. That's basically what I am accounting for and communicating when I report my very narrow confidence interval for where the population statistic of interest really lies.

Now if we walk backwards from there, of course, the confidence starts to decrease, and thus the interval of plausible population values - no matter where that interval lies on the number line - starts to widen. My sample is still deterministic as always, and I can calculate sample means and correlations, and I can treat those statistics as if they are claims about what I would be calculating if I had complete data on the population, but the smaller the sample, the more skeptical I need to be about those claims, and the more credence I need to give to the possibility that what I would really see in population data would be way off what I see in this sample. So all this is to sort of answer your question in reverse: our estimates of any out-of-sample statistics get more confident and converge on a single point, representing certain knowledge with complete data, for the same reason that they become less certain and range more widely the less data we have.

It's also important to understand that the standard deviation of a statistic specifically refers to and quantifies the probabilities of getting different sample statistics in different samples all randomly drawn from the same population, which, again, itself has just one true value for that statistic of interest. There is no standard deviation of that statistic at all in the population itself - it's a constant number and doesn't vary. A variable, on the other hand, has a standard deviation all its own, both in the population and in any given sample, and then there's the estimate of that population standard deviation that you can make given the known standard deviation of that variable within a given sample of a given size. So it's important to keep all the references straight, when you can have a standard deviation (or rather, a standard error) around a point estimate of a population variable's standard deviation, based off the standard deviation of that variable in your sample. There's just no simpler way to talk about it.

And lastly, note that, yes, it is certainly possible for a sample to give you a biased representation of the variances in the population, so, while it's relatively unlikely, it is always possible that a smaller sample will not just lie to you about the population statistic of interest but also lie to you about how much you should expect that statistic of interest to vary from sample to sample. There's no way around that. Think of it like if someone makes a claim and then you ask them if they're lying. Maybe they say yes, in which case you can be sure that they're not telling you anything worth considering. But if they say no, you're kinda back at square one. Either they're lying or they're not, and if you have no one else to ask, you just have to choose whether or not to believe them. (Bayesians seem to think they have some better way to make that decision but I humbly disagree.)