I have the following data: I have participants who answered on their favorite out of 6 positions of various information types on a display (e.g. 1 = bottom left, 2 = top left etc.).

I can now say for example the information “CPU status” was preferred to be shown on the top left by most of the participants, i.e. I can report the frequencies of the answers. But how can I test if these frequencies are not due to sampling or chance? In other words, how can i say that position a was significantly more often chosen than position b and c?

My only idea is to do a chi squared test with the two most frequent positions per information, but this seems cumbersome to me.

  • $\begingroup$ Why only the two most frequent? $\endgroup$ – nico Jul 7 '13 at 16:11
  • $\begingroup$ Is the null hypothesis that there are no underlying preferences among position, or something else? $\endgroup$ – Glen_b -Reinstate Monica Jul 7 '13 at 22:47
  • $\begingroup$ @nico: the two most frequent because otherweise i only know that there is a difference, but not between what answers. if i check only the two most frequent answers, i can say that is is significantly more often chosen than the second most frequent and thereby show that is is indeed the most frequent one?! $\endgroup$ – duplex Jul 8 '13 at 7:09
  • $\begingroup$ @Glen_b: there is no hypothesis that one position should be preferred! $\endgroup$ – duplex Jul 8 '13 at 7:10
  • $\begingroup$ @duplex: but then you may miss important information, such as cases in which people don't have a specific preference for 5 position but really hate the other one... Anyway, a while ago I asked a question about mxn contingency tables, maybe it could be of help? stats.stackexchange.com/questions/961/… $\endgroup$ – nico Jul 8 '13 at 7:28

A problem is knowing what a good underlying model should be when you focus on the most common choice. After all, there will usually be a most common choice.

You might find the analysis of Wilkinson (2006) of interest.

Wilkinson (2006) briefly reviews Pareto charts which commonly combine two displays in one. Frequencies in various categories are shown by a series of bars arranged in frequency order, from most common downwards. On that is often superimposed a rising curve showing cumulative frequency. Frequency and cumulative frequency may or may not have consistent scales. Examples from quality management studies often show categories of accidents, complaints, defects, failures, rejects, returns, or other such unwelcome phenomena. Wilkinson gives several cogent criticisms of this design and suggests an alternative: show frequencies in order, but by a dot plot, but add as reference a series of acceptance intervals.

These acceptance intervals are perhaps pertinent to your problem. They are calculated by simulation. Imagine as benchmark a population in which $k$ categories are equally probable, and imagine taking samples of size $n$. Just by chance the observed frequencies of the $k$ categories will typically differ. For each sample we can label the frequencies $f_{(1)} \ge f_{(2)} \ge \cdots \ge f_{(k-1)} \ge f_{(k)}$: thus $f_{(1)}$ is the frequency of the most abundant category, and so forth. Across several samples we can get order statistics for each $f_{(j)}$ and use those to calculate intervals with desired coverage.

The acceptance intervals should not be over-interpreted, for various different reasons.

First, the reference distribution is just an agnostic guess assuming that we know just the numbers of values and categories and that we have no reason to suppose that categories differ in probability. More commonly, we would not really expect that the categories are equal in probability; it is just that we would rarely know how to make our expectations precise.

Second, although making the sample size bigger will stabilize results, some variability will always be experienced in intervals produced by simulation.

Third, there are various slightly different recipes for producing percent points from order statistics.

A side-comment is that there appears to be no evidence that Pareto used what is now known as the Pareto chart, which seems to have emerged in quality management after 1951. See Wilkinson (2006) for more on the latter point.

Wilkinson, Leland. 2006. Revising the Pareto chart. American Statistician 60(4): 332-334.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.