In a survey conducted, 75 survey participants were asked to rate randomized keywords from groups 1-5. Every keyword belongs to a group and subjects were asked to rate each keyword. I would like to test the hypothesis the means of each group are equal but am unsure of the appropriate test to perform.

The data looks like this and there are about 8000 keyword observations.

anon    keyword         rating  is_in_group1    is_in_group2    is_in_group3    is_in_group4    is_in_group5
65161   perspectives    3              0    1   0   0   0
65161   read            4              0    0   1   0   0
65161   opportunity     1              0    0   1   0   0
65161   feature         4              0    0   0   1   0
65161   requirement     5              1    1   0   0   0

ANOVA / Kruskal-Wallis has an assumption of independence between groups, but if I have a list of each group1_ratings to group5_ratings, these groups would not be independent because the same user was asked to rate multiple keywords associated with each group correct? So the data is "paired" in a sense?

Also another factor is that the observations within groups are almost but not quite independent, e.g. it is possible for a keyword to be in both group1 and group3 or group1 and group2. How should this be accounted for as well?

What would be the best way to proceed with testing the hypothesis? Thank you!

  • $\begingroup$ In order to provide some context, could you give definitions of the five groups? (Why could you possibly expect them to be equally 'popular'?) Is the 'mean' of a group the proportion of the 20 words for which it was selected? I understand that the total number of $1$'s in your $20 \times 5$ data matrix exceeds $20,$ but what is that total? $\endgroup$ – BruceET Feb 23 at 1:21
  • $\begingroup$ I updated the problem with more context - there are a lot more observations than 20 and I tried to provide some more details on what the groups represent. It's sort of difficult to summarize (I can clarify more if it would help) but the same original questions hold with respect to what test would be valid given that the relationship between the groups might not be entirely be independent because each participant rated multiple keywords in each of the groups? $\endgroup$ – Matt Feb 23 at 4:19

Disclaimer: This was an attempt to answer a version of the Question which no longer exists.

Chi-squared GOF test. If you have $n$ subjects picking (single) categories for $n$ words, then you could use a chi-squared goodness-of-fit (GOF) test. The null hypothesis $H_0$ would be that each group is equally likely to be chosen: $p_0 = (1/5, 1/5, 1/5, 1/5, 1/5).$ The alternative would be that all groups are not equally likely.

If there are $X_i$ out of $n$ choices for Group $i,$ then the test statistic is

$$Q = \sum_{i=1}^5 \frac{(X_i - E)^2}{E},$$ where $E = n/5$ and $Q \stackrel {aprx}{\sim} \mathsf{Chisq}(\text{df}= 4)$ when $n \ge 25$ so that $E \ge 5.$

The critical value of this test at level $\alpha = 0.05,$ is $c = 9.488,$ for $n = 5$ groups and $n - 1 = 4$ degrees of freedom. (Computation in R statistical software.)

qchisq(.95, 4)
[1] 9.487729

Power and sample size. This GOF test has the advantage that one can find the power against various specific alternatives (in which some groups are more likely to be chosen than others). Let the alternative probabilities be $p_i,$ where $i = 1, 2, 3, 4, 5,$ and $\sum_i p_i = 1.$ Then the power is found using a non-central chi-squared distribution with non-centrality parameter $\lambda = n\sum_i (p_i - p_0)^2/p_0.$ The 'power' of the test is the probability of rejecting $H_0$ when it is incorrect in a particular way. In particular, suppose the alternative probabilities are $p_a = (2/6, 1/6, 1/6, 1/6, 1/6).$ (That is, Group 1 is twice as likely to be chosen than the others.) Then, for $n=20$ we have $\lambda = 2.7778.$

p.0 = 1/5;  p.a = c(2/6, 1/6, 1/6, 1/6, 1/6);  n = 20
lam = n*sum((p.a-p.0)^2/p.0);  lam
[1] 2.222222

Finally, the power of a test at level 5% is only about 0.18. So you would not be likely to detect even this marked departure from uniform group probabilities with $n = 20$ subjects.

c = 9.488; 1 - pchisq(c, 4, lam)
[1] 0.1871584

An analogous computation with $n = 200$ subjects gives power about 0.98, so you would be nearly certain to detect this departure from equal group probabilities. if present.

Notes: (1) Of course, there is a considerable advantage to planning an experiment according to a particular experimental design, so that one can have specific tests in mind from the start--and have a realistic idea how many subjects would be required in order successfully to detect important effects.

(2) There are also risks involved in analyzing data according to ad hoc procedures. Many datasets exhibit chance anomalies. Even if nothing of importance is truly supported by the data, one of very many ad hoc tests may randomly stumble across the '5% goal line'. Then the literature may be burdened by yet one more irreproducible result.

  • $\begingroup$ Thank you for your response! Sorry if this wasn't clear but the subjects weren't asked to select groups for every keyword, every keyword already belongs to a group and subjects were asked to rate each keyword. So I don't think I'm testing the null hypothesis that each group is equally likely to be chosen but rather are the differences in average ratings between the groups statistically significant? $\endgroup$ – Matt Feb 23 at 6:51
  • $\begingroup$ Oh well, then maybe the most important part of my answer is in the two notes at the end. // Perhaps you can edit some of the information still residing only in Comments into a clarified version of your Question. $\endgroup$ – BruceET Feb 23 at 7:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.