How sampling works

Question

Rattling around in my head I am trying to work out a little theoretical problem. Imagine we are doing a poll/study to find out something or other from the people across 3 cities.

We want as representative a sample as possible.

City A has 200,000 people, City B has 250,000 and City C has 50,000.

Going purely off that it seems easy to work out what % of people from each city you need to look at. 40% of A, 50% of B and 10% of C.

However. Then other factors come into play.

First is gender. Let’s assume for the sake of simplicity all cities have a 50-50ish split.

But then there could be other factors where the split isn’t so nice…For example age. Let’s simplify it into old, middle and young

            A   B   C
-Young      33% 20% 50%
-Medium     34% 60% 20%
-Old        33% 20% 30% 

As you can see here things get tricky with A being quite an equal spread but B being bulgy in the middle whilst C is presumably a (very extreme) university city.

In this problem…just what would you be aiming for to get a representative sample of the populations of the little 3 city region?

Further from that- just what maths are involved here for working it out? How does one layer ever more and more factors to work out samples?

Presumably in an attempt to get an ever better sample things could then look into job category, wealth level, voting inclination, etc…..

My worry here is that one could end up getting overall samples of each different factor and thus end up with a 50-50 gender split but with 90% of the women coming from city A just because they’re the ones who happened to respond.

Can anyone give a good guide as to just how sampling on this detailed a level works?

I am given a simplified example here and I am actually thinking on a much bigger level. I'm curious about just how representative polls on national or even global scales do/could work.

Answer 1

One of the first things to recognize is that you don't try to create a completely representative sample. There are way too many factors for a sample to be "completely" representative. As you say, there are age, gender, job category, wealth categorization, etc. To take it to an illogical extreme as an example, you could even further categorize your age - instead of "young, medium, old" you can do "18-15, 26-30, 31-35, etc." or even "18 year olds, 19 year olds, 20 year olds, etc.". This is obviously impractical for a whole lot of reasons and if you combine in things like job category, wealth category, etc. it becomes impossible.

So instead what you do is pick carefully the factors that you're concerned about. You never get a "completely" representative sample without sampling everyone, so instead you limit your factors (which also makes the analysis process more manageable). The experimenter will choose factors that they believe, based on assumptions made or previous experiments/surveys, will affect the outcome. Let's say we limit that to gender, your three age categories and city.

Once you've selected your factors, you then have two options. You can either use a proportional sampling design or weight calibration. Proportional sampling is when you specifically pick your sample to be representative of your population based on the factors that you are concerned about. So if City A has an even split of ages and also an even split of gender within each age group, you'd pick a sample that was balanced among the six categories (young-male, young-female, medium-male, medium-female, etc.) and accounted for 40% of the total sample between the three cities.

Weight calibration is a post-sampling adjustment, or weighting, of the importance of each group in the final calculations. So let's say you collect your sample and discover that in City A, 75% of your respondents were female and only 25% male, but the gender split is equal. You would give the female responses less weight in the final calculation and the male responses more weight, so that they contributed equally to the calculations.

You can use both - if you use proportional sampling and discover that your sample was not actually proportional due to non-responses, you can apply weight calibration to bring it back into balance.

A lot of national polls use weight calibration to help make sure that their polls are representative.

Answer 2

You are describing a fairly complex design - I'd caution you to be as parsimonious as possible. Remember that you will need to test for differences between all of your strata (you are, essentially, assuming that they are different populations, and testing for differences between strata is a requirement of this sampling design).

That said, you are describing some version of stratified random sampling. I suggest you look into the SamplingStrata package http://www.jstatsoft.org/v61/i04/paper.

I believe it is important that you ask this question before proceeding: If you were to randomly sample, what subsets of the population would be poorly represented? In this case, to determine which strata would be "poorly represented", you may want to attempt a power analysis to see the minimum number of samples required to detect an effect.

Cheers, and good luck.