Is my sample representative of the population I am trying to measure?

Question

We have completed a survey of FTSE350 companies, but only received 49 responses. We assume that this response rate means that our results are not representative of the FTSE350 as a whole, but wanted to show this more scientifically.

I'm sure this is a very simple calculation, but could someone please talk me through it. Thanks!

Answer 1

This problem is far from simple. My reply is a bit broader than your question.

1. It is not impossible that what you have is a representative sample, but in general you have no means of checking that or of measuring how unrepresentative the sample might be. Unfortunately, most other experience is that poor response to surveys goes hand in hand with biased response, namely what you have is not "representative", but you can't say exactly how.

2. In your case, there is a possible partial exception. You should be able to use published data for all the companies and get some indications of where your sample lies in that population. Naturally it's all too likely that the variables published are not those you wanted to measure; otherwise you wouldn't have wanted or needed to do a survey. Here, however, some information should be better than none in putting your sample in context.

3. The term "representative sample" is used a lot in non-technical literature, but that does not make it well-defined or easy to work with. In 1979 and 1980 W.H. Kruskal and F. Mosteller published a long series of articles in the International Statistical Review which remain valuable. The difficulties start with the trite but crucial: ensuring or checking that a sample is representative in one sense does no more than that. For example, you might find it easy to balance males and females in a sample, but balancing a sample according to say the income patterns of the population is enormously more difficult. See #2 again.

4. People who know a lot about compensating for poor sample response work for polling organisations or census bureaus, but their protocols, as I understand it, tend to be complicated, largely or entirely unpublished and partly ad hoc as they learn from experience.

5. Imputing missings from the rest of the dataset is a much used strategy in statistical science, but imputing missings when most of the data are missing and you have a biased sample is next door in difficulty to having no data at all.