Measuring representativeness of a (non-random) selection

Question

I am looking for a numerical value that expresses how representative a (non-random in this case) sample from a population is regarding the distribution of a certain attribute.

Take for example, the age distribution in a country and in its elected body of representatives.

I'll want to compare this value across many dimensions, so the value needs to be normalized in some way and not depend on the nature of the dimension. Some of the dimensions will be numerical and continuous (age, income), others discrete with many (county of residence, trained profession) or with few classes (gender).

The goal is to figure out in which of these dimensions the sample is most (or least) representative, i.e. in which ways the non-random selection (the electoral process) is biased.

Notes: There is a question of almost the same title here already, but I think it talks about something else, and this other question seems to ask for a very similar thing too, but there is no good answer for me there, either.

Answer 1

One way of posing these types of questions involves thinking about the group of representatives as if they had been selected by random sampling from the larger population. Your questions may then be quantified in terms of statistical likelihoods.

For example, the U.S. Senate currently includes exactly one African-American member, Tim Scott of South Carolina. African-Americans are 13.1% of the U.S. population. Your question could be formalized: "What is the likelihood that a random sample of size one-hundred from this population would contain fewer than two African-Americans?" This is an answerable question. The answer involves the probability mass function of the Binomial distribution: the probability that $n$ i.i.d. Bernoulli trials each with Bernoulli parameter $\theta$ will yield exactly $k$ successes is given by

$\Pr(K = k) = {n\choose k}\theta^k(1-\theta)^{n-k}$ .

For the present question, $n=100$ and $\theta = 0.131$. The probability that the number $K$ of "successes" is less than two is given by

$Pr[K=0] + P[K=1] = (0.869)^{100} + 100 (0.131)(0.869)^{99} \approx 1.283 × 10^{-5}$

or 0.001283%. Expressed as an odds ratio, the chance that an unbiased selection procedure would produce such a small number of African-American senators is 77,960 to 1.

Of course, U.S. senators are not elected nationally, they are elected state-by-state, so a more careful analysis would involve the state-by-state distribution of populations by ethnicity. But you get the idea.

Based on your summary description of your problem, it appears that the answers to your other questions could likewise be quantified in terms of probabilities, in a like manner.

(If you wanted to, you could go a step further and pose your questions in terms of hypothesis testing. I'm not a big fan of hypothesis testing, but some people seem to like it.)