# Measuring representativeness of a (non-random) selection

I am looking for a numerical value that expresses how representative a (non-random in this case) sample from a population is regarding the distribution of a certain attribute.

Take for example, the age distribution in a country and in its elected body of representatives.

I'll want to compare this value across many dimensions, so the value needs to be normalized in some way and not depend on the nature of the dimension. Some of the dimensions will be numerical and continuous (age, income), others discrete with many (county of residence, trained profession) or with few classes (gender).

The goal is to figure out in which of these dimensions the sample is most (or least) representative, i.e. in which ways the non-random selection (the electoral process) is biased.

Notes: There is a question of almost the same title here already, but I think it talks about something else, and this other question seems to ask for a very similar thing too, but there is no good answer for me there, either.

-

One way of posing these types of questions involves thinking about the group of representatives as if they had been selected by random sampling from the larger population. Your questions may then be quantified in terms of statistical likelihoods.

For example, the U.S. Senate currently includes exactly one African-American member, Tim Scott of South Carolina. African-Americans are 13.1% of the U.S. population. Your question could be formalized: "What is the likelihood that a random sample of size one-hundred from this population would contain fewer than two African-Americans?" This is an answerable question. The answer involves the probability mass function of the Binomial distribution: the probability that $n$ i.i.d. Bernoulli trials each with Bernoulli parameter $\theta$ will yield exactly $k$ successes is given by

$\Pr(K = k) = {n\choose k}\theta^k(1-\theta)^{n-k}$ .

For the present question, $n=100$ and $\theta = 0.131$. The probability that the number $K$ of "successes" is less than two is given by

$Pr[K=0] + P[K=1] = (0.869)^{100} + 100 (0.131)(0.869)^{99} \approx 1.283 × 10^{-5}$

or 0.001283%. Expressed as an odds ratio, the chance that an unbiased selection procedure would produce such a small number of African-American senators is 77,960 to 1.

Of course, U.S. senators are not elected nationally, they are elected state-by-state, so a more careful analysis would involve the state-by-state distribution of populations by ethnicity. But you get the idea.

Based on your summary description of your problem, it appears that the answers to your other questions could likewise be quantified in terms of probabilities, in a like manner.

(If you wanted to, you could go a step further and pose your questions in terms of hypothesis testing. I'm not a big fan of hypothesis testing, but some people seem to like it.)

-
There is a noteworthy economic literature on statistical measurement of race and sex discrimination in, e.g., labor markets, that might be useful. A seminal work is: The Statistical Theory of Racism and Sexism Edmund S. Phelps The American Economic Review Vol. 62, No. 4 (Sep., 1972), pp. 659-661 Published by: American Economic Association Article Stable URL: jstor.org/stable/1806107 – Arthur Small Jan 17 '13 at 17:05
Great answer, exactly what I was looking for. For completeness, can you also provide a link to hypothesis testing and why you don't like it? – Thilo Jan 20 '13 at 1:37
@Thilo: Oh, dear. I suppose I invited that follow-up, so have only myself to blame. Answering would, alas, take a long while. 'Any' stats textbook will cover the topic: see, e.g., W.H. Greene, Econometric Analysis. On 'why not?', I'll recommend a book I find canonical that covers the philosophical and practical problems of hypothesis testing quite well and in depth: Statistical Decision Theory and Bayesian Analysis by J.O. Berger, amazon.com/gp/product/1441930744. Very short answer: I'm interested in using stats to make better decisions under uncertainty. HT doesn't help. – Arthur Small Jan 20 '13 at 2:22