I generated a list of 35 proven benefits on a subject and asked the public to choose five which would be most likely to influence their behaviour.

43 respondents cast 215 votes, and three of the benefits received many more votes than the others.

What would be the best statistical tool to measure the significance of this result?

Clarifications: The research has been motivated by the practical difficulty of promoting all 35 benefits to the public, in order to generate a change in behaviour. The research aim was to make an initial determination whether each benefit was likely to have an equal influence on the public. Before seeing the responses I expected a fairly even distribution across all benefits but this was not the case. The top three benefits were at least 10 votes clear of the fourth most popular. Four benefits received no votes at all. I hope to demonstrate the pattern of response is statistically significant for x number of the benefits, and that future promotional material should focus on these as the most likely benefits to generate change.

Each of the benefits is distinct from the other, though the first 13 would come under the category of environmental, then 9 economic and 13 social, and it would be desirable to consider the data by these groups in addition to considering each individual benefit.

The respondents selected the five benefits from the list of 35 that were most likely to influence a change in their behaviour, they did not rank their five choices and successive votes did not affect previous ones, other than you could only vote for each benefit once.

Many thanks for the initial responses and prompts to clarify and I hope I have remembered everything!

  • 2
    $\begingroup$ It's extremely difficult to measure the significance of results that are noted after the fact. What, precisely, would constitute a "result"? Having at least one benefit with many more votes than the others? Having exactly three such benefits? Having some number of benefits (less than 35) with many more votes than the rest? Having three benefits with approximately the same numbers of votes? Etc., etc. For this reason, you will get the strongest answers by stating your original research aim. That is, before you saw the data, how would you have characterized an interesting result? $\endgroup$ – whuber Aug 5 '11 at 16:29

We can take a tip from the literature on outlier detection. In effect, this dataset can be summarized as a sequence of counts of the 35 benefits. The null hypothesis is that the counts were generated by a process in which individuals chose benefits completely at random and independently of one another. The alternative you want to be able to detect is that some previously unknown number $r \ge 0$ of the counts are chosen more frequently than the rest.

Good tests for this situation (of an unknown number of high outliers) often rely on gaps, and that's precisely the statistic highlighted in the clarification to the question: "The top three benefits were at least 10 votes clear of the fourth most popular." But, to be thoroughly honest in our analysis, let's look at the problem as if we did not know the results yet.

A priori, then, it looks like identifying an unusually large gap among the highest counts might do a good job of separating counts that are too high to be explained by chance. But another alternative may also be of interest: it could be that each benefit (as indexed, say, by $1 \le i \le 35$) is chosen with probability $p_i$ and that there is a slight but important amount of variation among the $p_i$. This could lead to a graduated series of elevated counts: e.g., one benefit might be chosen $19$ times, another $18$ times, another $17$ times, ..., etc, thereby exhibiting no unusual gaps in the counts. It would be good to detect this situation, too.

I am thereby led to propose a two-criterion test based on a pair of statistics: the highest count and the largest gap among the highest half of all the counts (because we're not interested in any low outliers).

The crucial thing is to establish the joint distribution of these statistics under the null hypothesis (to wit, $p_1 = p_2 = \cdots = p_{35}$). That we can readily do with simulation. The following results are each based on 1,000,000 independent replications of the experiment; each set of results is derived from a separate set of such replications.

Null distribution of the largest gaps

Histogram of the max gap frequencies

The estimated complementary cumulative distribution function (good to 3-5 decimal places based on this simulation) is

Max gap   p
1         1.000
2         0.442
3         0.136
4         0.0434
5         0.0135
6         0.0039
7         0.00011
8         0.00003
9        <0.00001

Thus, if you wished to use the max gap test to identify high outliers at the $\alpha=5\%$ level, you would read down this chart until $p$ first fell below $5\%$, finding the critical value of $4$. According to this test, if the largest gap among the top half of the counts (sorted from smallest to largest) were $4$ or greater, you would conclude that some benefits are selected significantly more frequently than others, and those would be all those with counts occurring above the lowest gap of $4$ or more occurring among the top half of all counts.

For example, one trial resulted in the following:

  • Three benefits were each selected by 3 people
  • Five benefits were each selected by 4 people
  • Four ... 5
  • Eleven ... 6
  • Three ... 7
  • Seven ... 8
  • One ... 9
  • One benefit was selected by 14 people.

We could equally well say that the ordered set of counts is

14, 9, 8, 8, 8, 8, 8, 8, 8, 7, ..., 4, 3, 3, 3.

(As a check, $3*3 + 5*4 + 4*5 + 11*6 + 3*7 + 7*8 + 1*9 + 1*14 = 215$ accounts for all $5*43$ choices.) In this trial, the upper half of the counts comprise those $6$ or greater. The nonzero gaps, starting at $6$, are $7-6=1$, $8-7=1$, $9-8=1$, and $14-9=5$. The last is the largest gap and only the count of $14$ exceeds it. We conclude (a) there is at least one benefit chosen significantly more than the rest and (b) it consists of just the one benefit that was chosen by $14$ people.

In another case, two benefits were each selected by $9$ people, one benefit by $14$ people, and one benefit by $15$ people. Here the maximum gap was $k=14-9=5$ and two benefits are identified as chosen significantly more than the rest.

In yet another case, the ordered set of counts was

21, 15, 15, 14, 14, 14, 10, 10, 9, ... .

The max gap is $21-15=6$. However, there's another gap of at least $4$ among the highest counts: $14-10=4$. Therefore, the six highest counts would be considered elevated (to wit, $21, 15, 15, 14, 14, 14$) at the $5\%$ level.

Conditional distribution of the highest frequencies

In case the maximum gap test fails to detect any significantly high choice, it is still possible that many choices may have been selected more frequently than on average. Our supplemental test for this situation looks at the highest frequency. To investigate this, we need to evaluate the joint distribution of the maximum gap and the highest frequency when the maximum gap is less than the critical value of the maximum gap test.

Here is a plot of the joint distribution under the null hypothesis (at least where the probabilities are visibly greater than zero). For a better view, it is shown with axes reversed: larger values of both statistics occur near the front.

Joint distribution

As we have seen, in many applications the critical value will be either $4$ (for $\alpha \approx 5\%$) or $5$ (for $\alpha = 1.35\%$) or possibly $6$ (for $\alpha \approx 0.4\%$). We would then only need to inspect the maximum frequency when the maximum gap is less than or equal to $3$ (or $4$ or $5$, respectively). Here is a detail of the joint distribution plot for this region:

Join distribution detail

Suppose, for example, we still want to conduct a test at the $\alpha=5\%$ level. We saw above that this indicates a critical threshold of $4$ for the max gap test. The actual level achieved by this test is $4.34\%$. This leaves a little room to screen by maximum frequency. The simulation shows that when the maximum gap is less than $4$, we can also include any dataset whose maximum frequency is $15$ or larger within the critical region. When the null hypothesis is true, this joint test statistic lies in the critical region $4.73\%$ of the time: comfortably close to the desired $5\%$ level. (If we were more aggressively to include a maximum frequency of $14$ within the critical region, $\alpha$ would jump to about $7.7\%$, too far above the targeted value of $5\%$.)

For example, suppose the highest counts are $15, 14, 12, 10, 8, 8,$ etc, with a maximum gap of $14-12=2$. This would not be identified as significant with the max gap test (at $\alpha \approx 5\%$), but the maximum value of $15$ is evidence of some tendency to select at least one (but some unknown) number of benefits more often.

Other good combinations for critical values of (max gap, max frequency) are:

(4, 15): 4.7% 
(5, 16): 1.5%   (5, 15): 2.4%
(6, 17): 0.4%   (6, 16): 0.7%
(7, 18): 0.1%   (7, 17): 0.2%

The second instance in each row is more sensitive to slight differences in the $p_i$ while the first instance primarily tests for identifiable high outliers, which is probably what is most desirable here.

Obviously in any similar situation where these parameters (43, 5, 35) are different, the simulations would have to be re-run accordingly. Simulations are needed, by the way, because these parameters are small enough that neither a Poisson approximation nor approximating by a collection of iid Binomials is sufficiently accurate for calculations. Once 5*35 = 215 gets larger and 43 * 5/35 = 6 gets smaller, these approximations can fruitfully replace simulations.

Now let's analyze the data. Reportedly there is a gap of $10$ between the three highest counts and the rest. This identifies the three highest counts as significantly elevated. It's possible that additional gaps of $4$ or greater occur. For instance, the ordered set of counts might be

28, 28, 25, 15, 14, 10, ... .

In this case, if we were testing at the $5\%$ level, we would also identify the two benefits chosen by $15$ and $14$ people as being significantly elevated compared to the rest, because their counts both lie above a gap of $14-10=4$, giving five elevated benefits in toto. If we were testing at the $1\%$ level these two benefits would not be significant, leaving just three elevated benefits. If we were testing at the $2.5\%$ level with critical values of $(5,15)$ we would still only identify three significant benefits: the max frequency test is used only when the max gap is less than its critical value (of $5$).

A note about implementation

The calculations and illustrations presented here were done in Mathematica (8.0). Its language makes a good pseudocode no matter what platform is intended to be used in the end.

The core function, trial, simulates a single experiment:

trial[respondents_, votes_, choices_] := 
       RandomSample[Range[choices], votes], {i, 1, respondents}]], {0, choices+1, 1}]]]

Each "respondent" takes a random sample of votes from the set 1..choices. The results are combined (Flatten) and tallied in integer-width bins (Rest[BinCounts..]) from 1 through choices. The counts are sorted for later analysis. For example, one call to trial[43,5,35] resulted in the array (2, 3, 3, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 6, 6, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 10), indicating that one choice was selected by two respondents, two other choices were selected by three respondents, etc.

The maximum gap above the median for any such trial t is computed as

maxGap[t_] := Max[Differences[Take[t, Floor[-(Length[t] + 1)/2]]]]

It's just the largest difference among the last (35+1)/2 = 18 elements in t.

A Monte-Carlo simulation of the max gap statistic can then be generated as

data = ParallelTable[maxGap[trial[43, 5, 35]], {i, 1, nTrials=1000000}];

The first illustration, Null Distribution of the Largest Gaps, was created in this way by making a histogram of the gap frequencies:

gapFreq = Sort[Tally[data], First[#1] <= First[#2] &];
 ChartLabels -> First[Transpose[gapFreq]], 
 AxesLabel -> {"Max gap", "Frequency"}]

The rest of the analysis similarly computes and displays properties of the simulated data.

  • $\begingroup$ Many thanks for this answer @whuber which is exactly what I was hoping for, and far exceeds my expectations! As I try to complete my analysis as you described I have failed miserably to replicate the simulation, graphs and estimated p values to use with the different groupings, no surprise to me! I have summarised the data and made it available (hopefully) using the following link Can you advise how I can reproduce the simulation for my All P group (37,5,35) and treat env,eco,soc groups? $\endgroup$ – Steve Aug 10 '11 at 9:53
  • $\begingroup$ @Steve I added some details of the simulation. You can use them directly in Mathematica or use them as a guide to implementing the simulation on some other platform. $\endgroup$ – whuber Aug 10 '11 at 12:42

One clarifying question: are you interested in the significance of the three top responses jointly, or each of them individually? This probably wouldn't affect which general method you choose, but would affect the test statistic.

Second clarification: how related are the responses? (I mention this below, too.) For instance, it could be the case that potential responses 1-5 are essentially identical ("lower price" and "costs less"), as an extreme example, and that 6-10, 11-15, etc., are also, in which case the data would really be 43 single votes for one of 7 options; or all 35 are completely different ("lower price", "dogs are loyal companions"); or different sized groupings of similar but not identical responses.

The first idea that came to mind for me was permutation tests. [Ultimately, I'm not sure this is a good idea, but I will leave what I originally wrote.] Philip Good's text is a good start; this link has the entire book in PDF format. The key assumption is "exchangeability" under the null hypothesis; see Section 2.2.2, page 24, and Appendix A.5, where he defines, "the joint distribution of the observations is invariant under permutations of the subscripts" (Good p. 268); more of my thoughts below. If that assumption is satisfied, permutation tests are exact, unbiased, nonparametric tests. The only other difficulty I can think of is computation. 35! is approximately 10 duodecillion...too big to compute every single permutation (again, unless there is some valid way to group responses?). Chapter 14 in the book, "Increasing Computational Efficiency," discusses seven techniques (it says); of course, the "exactness" would be lost, but might still be a good option. I think the more critical piece is exchangeability.

Re: exchangeability, I think for you it would depend on how related the responses are to each other. For your setup, it would be the indices of the 35 potential responses that you would want "exchangeable." If there are clumps that are very similar, maybe you could group them somehow...? Even if everyone had only voted once, and you can assume the people are independently sampled, if response options have some dependence structure, the results wouldn't be exchangeable. It only gets trickier if the distribution of my second vote depends on my first vote, but doesn't depend on the previous person's first vote.

So the second idea would be some sort of bootstrap. You could resample at the unit of the person (43 in your sample), which I think would take care of the intra-person vote dependence. Your test statistic would either combine the three categories you mentioned, or treat one at a time. Perhaps a way to get a pivotal statistic, but I can't think of one right now.

Other than that, I'm not sure. If you assumed any one of the 35 were distributed binomial(215,1/35), I don't think that would quite be right given the dependence issues. So I'm curious what others will suggest for you, and the best I can suggest meanwhile is a bootstrap at the person level.


  • $\begingroup$ Thank you for the prompts to clarify whuber and @Davidm, I have edited the initial question to add information. The Good text is fantastic, thank you for the link. $\endgroup$ – Steve Aug 5 '11 at 17:35
  • 1
    $\begingroup$ Steve, glad you like the text. It sounds like your assumptions under the null hypothesis fit well with a "permutation test" like whuber's answer (which uses simulation to get around the computational challenge--best option I think). Depending on the categories of the top 3 responses, tests on the 3 categories (env, econ, soc) might be interesting too (like you said), in the same spirit as whuber--what's the distribution if everyone votes randomly, vs. how did everyone actually vote? $\endgroup$ – David M Kaplan Aug 9 '11 at 15:11

Since you were asking your respondents to make a choice, you would want to look at literature on choice modeling. This has been developed in econometrics to a far greater extent than in any other discipline (and, interestingly, in transportation research, which has, to a great extent, been relying on econometricians pushing the methodology). I believe you have a weird twist on a multinomial logistic model or a conditional logistic model. If you find an appropriate extension (five selected alternatives rather than one, as in the standard multinomial logit), and you fit an intercepts only model, it will tell you by how much some alternatives are more likely to be chosen than others. The framework will also allow you to have covariates, so you would also be able to dissect how the demographics of your respondents affects the choices they make.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.