# How to sample when you don't know the distribution

I'm fairly new to statistics (a handful of beginner-level Uni courses) and was wondering about sampling from unknown distributions. Specifically, if you have no idea about the underlying distribution, is there any way to "guarantee" that you get a representative sample?

Example to illustrate: say you're trying to figure out the global distribution of wealth. For any given individual, you can somehow find out their exact wealth; but you can't "sample" every single person on Earth. So, let's say you sample n = 1000 people at random.

1. If your sample didn't include Bill Gates, you might think there are no billionaires in existence.

2. If you sample did include Bill Gates, you might think billionaires are more common than they actually are.

In either case, you can't really tell how common or rare billionaires are; you may not be even able to tell whether any exist at all.

Does a better sampling mechanism exist for a case like this?

How would you tell a priori what sampling procedure to use (and how many samples are needed)?

It seems to me that you might have to "sample" a huge percentage of the population to know, with anything approaching reasonable certainty, how common or rare billionaires are on the planet, and that this is due to the underlying distribution being a bit difficult to work with.

• In the distribution of wealth case, much would depend on what exactly the aim was. If for example the aim was to estimate the levels of wealth that would put an individual into the top 10%, top 20%, and so on, then it wouldn't be critical whether or not the sample included billionaires. But if the aim was to estimate the proportion of wealth held in total by the top 10%, then how the sampling handled billionaires probably would be critical. The general point here is that whether a sample is representative is always relative to what you are trying to do. Oct 24 '14 at 17:42
• really? open problem, answers being good, still are approximations (sometimes better, sometimes worse). It is an open problem, maybe the single open problem of statistics Oct 31 '14 at 7:53

I dispute your claim that "In either case, you can't really tell how common or rare billionaires are". Let $f$ be the unknown fraction of billionaires in the population. With a uniform prior on $f$, the posterior distribution of $f$ after $1000$ draws that turned out to have 0 billionaires is a Beta(1,1001) distribution, which looks like this:

While the posterior distribution of $f$ after $1000$ draws that turned out to have 1 billionaire is a Beta(2,1000) distribution, which looks like this:

In both cases, you can be quite certain that $f < 0.01$. You might think that isn't precise enough. But actually 0.01 is quite precise for a sample of size 1000. Most other quantities that you might estimate would be less precise than this. For example, the fraction of males could only be estimated within a range of size 0.1.

There are two things you could do (separately or in combination)

Model the tail

One is to model the tail of the distribution using a parametric distribution. Power laws are known to fit the distribution of wealth well, so you try a Pareto distribution. You would either fit that distribution by maximum likelihood, that is, by finding the parameters which best represent your sample. Or better, you could put a Bayesian priors on the parameters, and compute the full posterior.

Unfortunately, power laws are very sensitive to parameters, and without many large datapoints in your sample, there will be a lot of uncertainty about the exponent. The estimated number of billionaires will be sensitive to this parameter, but much less than the average wealth of billionaires, so the situation isn't too bad.

Importance sampling

The other one is to change the way you collect your sample. Suppose that you suspect (as you should) there are more billionaires per capita in Monaco or Zurich than in Mogadishiu. If you know the population of each of these cities, you could collect a larger sample in the cities where you expect to see more billionaires, and a smaller one in the others.

So say Zurich has 400,000 people and Mogadishu 1,400,000 and we want to poll 9,000 people. We're interested here in the number of millionaires, not billionaires.

A unbiased sample would select 2,000 people in Zurich and 7,000 in Mogadishu. However, we'll bias the sample by sampling seven fold more often from Zurich. So we'll "pretend" that Zurich has 2,800,000 people and adjust later. This means we'll poll 6,000 people in Zurich instead of 2,000 and 4,000 in Mogadishu.

Say we count 21 millionaires in our Zurich sample, and only 1 in our Mogadishu sample. Since we over sampled Zurich 7 fold, we would only count it as 3 millionaires.

This procedure will decrease the variance of your estimator. It can also be used in conjunction with the first method, in which case you will be adjusting for the importance sampling when fitting a parametric distribution.

I think a good sampling method is based on previous knowledge of the system. In your field, you have knowledge about potential biases that might affect your sampling. If you don't have that knowledge, you can acquire it from literature.

In your example, you know that there is billionaires and that they might bias your sampling. So you can decide to stratify the sampling by education level, country, type of job, etc. There are multiple options.

Let's try with another example. Your objective is to determine the abundance of a mice species in a park. In this park, there is forest and meadows. By the literature, you know that mice are more abundant in forest than meadows. So you stratify your sampling by this characteristic. There is other sampling procedure possible, but I think your best information will be from the existing literature.

And if there is no literature about your field ? Improbable, but in that context, I would do a pre-study to see what factors need to be taken into account for sampling.

Whether a sample is representative or not has nothing to do with the observed measurements of the sample. A sample is representative if every set of observational units has the same probability of being chosen as any other set of the same size. Of course this is hard to do unless you can get a complete enumeration of your sample space. Assuming you can get that (from census tract data, for instance), a simple random sample will be representative.

No matter how you obtain your sample, there will always be at least three separate sources of error to consider:

sampling error: by chance you include Bill Gates in your representative sample. Statistical methods, especially the widths of confidence intervals etc. are designed to take care of this, provided you have some rough knowledge of the distribution at hand (e.g. normality, which wealth distribution definitely does not possess).

sampling bias: The sample was not representative. Example: Bill Gates has an unlisted number, so your telephone survey could never reach him (unless you use something like "random-digit dialing"). This is an extreme example, but sampling bias is very widespread. A common occurrence is to take on-site or convenience samples: You sample restaurant patrons at the restaurant as to whether they like the place, how often they have been there, and whether they plan to return. Repeat customers are far more likely to be sampled than one-time customers, and samples of this type can be severely biased in their attitudes.

response bias: The measurements themselves are inaccurate. This can came about because of anything from malfunctions of the meter to conscious lying to quantum effects (e.g. Heisenberg's uncertainty principle).

• This answer has useful advice and covers good ground. I would like to suggest that the characterization of "representative" may be too restrictive, however, because it excludes common and useful forms of sampling (including some specifically mentioned in other answers) such as stratified sampling, importance sampling, and forms of systematic sampling. Wouldn't it suffice to allow that a sample is representative when the chance of including any set of observational units is known (and therefore can be used to produce unbiased estimates) but not necessarily constant for all sets of a given size?
– whuber
Oct 24 '14 at 19:21
• @whuber "Wouldn't it suffice to allow that a sample is representative when the chance of including any set of observational units is known...": This is correct, and I should edit my answer to acknowledge stratified sampling and importance sampling. However, systematic sampling is dicey, and the advice given in link and elsewhere is simply wrong. If there are systematic patterns in the data, a randomized starting point will not eliminate the bias, all it will do is to make sure you won't be able to compute the bias. Oct 26 '14 at 2:13
• maybe the best answer so far (in the sense of aiming directly to the statistical point) Oct 31 '14 at 7:56