2
$\begingroup$

Suppose the target population is a group of 11 company CEOs and 7 CEOs responded to a question about their level of annual bonus on a 7 point Likert scale.

The responses are as follows: 7, 7, 7, 7, 7, 7, and 5. The average response score then is 6.71.

If the last two levels of the Likert scale are '6 = satisfied' and '7 = very satisfied', this finding can be expressed as a 'very satisfied' rating.

Do we apply a margin of error to Likert scale responses?

(The margin of error for a sample of 7 is 38%. This is 1 divided by the square root of the sample - a rough guide!)

$\endgroup$
2
  • 1
    $\begingroup$ At bottom, the question is a great one: "What does an MoE mean for Likert scale responses?" May I point out, though, that (1) this question is discussed extensively in many other questions here; and (2) the rest of the question is confusing. For instance, it's mathematically impossible for an average of 7 integers to equal 6.67. Second, what does "38%" mean as a MoE for responses from 1-7? Third, where does this 38% value come from? Fourth, perhaps this question is really about missing values rather than about MoE's? (If missingness is not at random, it's hard to say much here...) $\endgroup$
    – whuber
    Jan 30, 2012 at 5:04
  • $\begingroup$ Depends what, if anything, you're trying to infer about the 11-7=4 CEOs who didn't respond; was that missing-at-random, "decline to participate", "could not be contacted", "not included in survey" or what? do we simply exclude those 4 when computing MoE? I assume you just want to exclude them and impute MoE for the 7 who responded, not build a model of all possible responses they might have given instead of NA, then factor that into a (very pessimistic) inferred MoE? $\endgroup$
    – smci
    Apr 21 at 3:43

1 Answer 1

6
$\begingroup$

The "margin of error" of 38% is computed using a formula for 0/1 results that are obtained independently and randomly from a large population. None of these apply. The analogous formula for the present case (with 1..7 results obtained from a small population assuming random non-response) would involve a sample standard deviation with a finite population correction. It would not be very helpful in sorting out the confusion. Instead, let's explore the concept starting from its foundations.

The purpose of a margin of error is to indicate how much the population might differ from the sample, assuming that the data are a random sample of the population.

The problem we have is we don't know what the four missing respondents would have said. We have to cover all possible cases that are consistent with the data.

An interesting possibility is that seven of the 11 CEOs would give a reply of 7, two of them would reply with 5, and the other 2 with 1: this is (hypothetically) the population. How consistent are the data with this scenario? In this case, the chance of observing six 7's and one 5 at random is

$$\frac{\binom{7}{6}\binom{2}{1}\binom{2}{0}}{\binom{11}{7}} = \frac{7 \times 2 \times 1}{330} \approx 4.24\%.$$

I imagine that observing seven 7's ($1/330$ chance) or five 7's and two 5's ($21/330$ chance) would also be considered a "very satisfied" overall rating. The total chance of observing this rating in the sample would therefore equal $(14 + 1 + 22)/330$ = $11.2\%$.

This is about the worst situation that is consistent with the recorded observations in the sense that the conclusion made from our sample ("very satisfied") has at least a 5% chance of arising from a random selection. A reasonable way to express the "margin of error," then, is to note that as many as two of the CEOs, but not more than that, might have been extremely dissatisfied.

It's a good idea to go further with the analysis, because we have no evidence to support the assumption that the seven respondents actually are a representative sample. In fact, most likely they are not. Perhaps, for example, four of the CEOs would reply with 1's but did not care to because they did not want to reveal their dissatisfaction: the missingness is not at random.

A frank and thoughtful exposition of the results would bring up both these points when assessing the credibility of the results and their applicability to the entire population. Its conclusions would necessarily be tentative. They might be stated thus:

In this self-selected sample of 7 of the 11 CEOs, six reported being "very satisfied" and one "somewhat satisfied" with their bonus. Nothing is known about what the remaining four CEOs feel. We can conclude that as a group, the majority of the 11 CEOs were highly satisfied, but there may be anywhere from none through a significant minority (four) who were dissatisfied.

One advantage of this plain exposition is that it makes no unnecessary technical demands on the reader (or the writer) by invoking a "margin of error" formula which would need to be explained and interpreted and might well be incorrect by not accounting for the small population size or possibility of non-random response.

The techniques illustrated here apply to similar problems with small population sizes, non-random responses, or qualitative ordinal scales.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.