Well, first, of course, it is only true when the sample observations are uncorrelated. Otherwise the sample mean converges on a number that is higher than the true mean by an amount that increases as the average covariance increases.

Secondly, many distributions, particularly those that have power law-like characteristics -- distributions that are sometimes called fat-tailed or leptokurtotic – do not have sample means converging on populations means. For example, the sample mean of the Pareto distribution generally has an expected value less than that of the true mean, by an amount that increases with tail thickness.

The most common reason why a larger sample may not be better is that the sampling procedure is biased. In this case, bigger samples only enable you to get a more precise estimate of the bigger samples will only enable you to be wrong more precisely.

The mean of correlated variables may, depending on the form of the correlation, not converge to the true mean as sample size increases. The weak law of large numbers as it is usually stated applies to iid samples. A variant of the law, due to Chebyshev, shows the weak law still applies to sequences that are not independent provided that the average covariance of the elements in a sequence goes to zero as the sample size increases. What is average covariance does not go to zero?

Here is a simple example. Suppose you think of your random variable as consisting of two components, the true mean u and an error term e sub i. Let e sub i = a*f sub i + (1-a)f, where each of f sub i and f represent the outcome of the flip of a fair coin, equal to 1 if heads, -1 if tails, and 'a' is a constant between 0 and 1. Suppose further that f sub i is a separate flip for each realization of the random variable, representing the uncorrelated part of the error, while f is based on the flip of a single coin for the entire series, and represents the correlated part. We can vary the correlation of the errors continuously between independence and perfect correlation using 'a' as a slider. The average of the uncorrelated part will go to zero as the sample size increases. The correlated part, on the other hand, does not go away. The uncorrelated part complies with the weak law and vanishes in the limit. The correlated part has an a priori mean of zero, but since it is common, increasing the sample size does not drive the sample mean toward the true mean, but only toward the sum of the true mean and the correlated portion of the error.

The mean of a sample from some skewed distributions with fat tails distributions, such as the Pareto for some parameter values, does not converge as the sample size increases in the way we expect, and is usually below the true mean. The Pareto distribution has, in the range of parameter values in which it is most often employed, a mean but no defined variance. The version of the weak law that you most often see requires that the distribution have both a mean and a variance, but the variance merely simplifies the proof and is not actually required. So we know – the weak law of large numbers proves – that the distribution of the mean of a sample drawn from a Pareto distribution converges in probability to the population mean.

However, as is generally true of skewed distributions, the instances of the sample mean are not evenly distributed around the true mean. For distributions with a right tail and no left tail, most of the sample means will lie below the true mean. However, you will see the occasional sample which contains a member from somewhere far out in the tail, with a mean well above the true mean, and these rare but large deviations draw the mean of sample means toward the true mean.

For distributions like the Pareto, the difference between the strong law and the weak law really bites. The weak law of large numbers guarantees only convergence in probability. This still allows arbitrarily large divergences which never disappear, so long as they become rarer as the sample size increases. And these rare large divergences of sample means in the upwards direction can greatly slow convergence toward the mean of samples at the bottom end of the distribution of sample means, and of the collection of all samples.

Take the following example, chosen based the original motivation in the development of the Pareto distribution. Wealth distribution and many other things are often described as having an 80:20 distribution, i.e. 20 percent of the population has 80 percent of the wealth, 20 percent of the customers make 80 percent of the purchases, etc. This is common enough that the 80:20 distribution is frequently called the “Pareto Law”, though Pareto did not make the claim that this particular value was universal, and indeed it is only approximately true. But it is approximately true of, e.g., the top one percent of U.S. wealth holders.

So let’s look at a Pareto distribution with a mean of 20 (in trillion USD) to represent approximately the combined wealth of the wealthiest one percent in the U.S., and a tail parameter of 1.16, to reflect the 80:20 law. (Note that this value is in the range (1<σ<2) for which the Pareto does not have a variance.) I fired up my R and produced 10,000 samples of size 100 drawn IID from this distribution, then 10,000 samples of size 1000, then 10,000 samples of size 1,000,000. Across all the runs of different sample sizes, samples had means that were below the true mean 80 to 86 percent of the time. No surprise there. However, the mean of 10,000 IID samples of size 1000 gave me an estimated wealth of only about 16 trillion, which was a lot lower – a lot further from the mean - than I would have guessed for a sample of this size. Moreover, on doing a histogram of the size 1000 samples, and I was surprised to observe one sample with a sample mean for estimated wealth of 3,000 trillion USD, i.e. 150 times the true value – and this was not enough to pull the average of the samples up to the true value.

Turning to the samples of a million, and treating the entire run as a single unified sample instead of a bunch of draws of a million, i.e. looking at a random sample of ten billion IID draws, the estimated mean was still only 17.5. In other words, we are seeing convergence in probability, but it is so slow that, over the range in which it is typically possible to vary the sample size by, e.g. collecting more data, it hardly occurs at all. This is nothing like the decline with the square root of sample size that we see with some other common distributions. Moreover, your typical or modal sample is substantially below the true mean, and this remains true even for very large sample sizes.

This suggests that you might not want to think of these fat-tailed distributions in the same way as you think of distributions with finite second, third, and fourth moments, and that the sample mean may not be the estimator of the true mean that you want to use in such cases.