I am trying to compare Likert-scale answers to a survey applied in 3 different cities: A, B and C. I arbitrarily chose twice as many respondents from A, for a total of 500 answered questionnaires in the following proportion: A 250 B 125 C 125

Nevertheless, these quantities are not in exact proportion with the actual populations of those cities, for which I am being asked to review my research. I am using a Kruskal-Wallis test to compare answers among the 3 groups, and for all 33 questions in the survey, the p-value is not significant.

Would I be right to argue then that although the sample is not representative of the underlying population, there does not appear to be a difference in responses from the 3 cities?

I have conducted cluster analysis and principal component analysis with these data, and I am being asked to include "weights" to fix the representation issue. My goal is not to have to do this and justify that there is no statistically significant difference in responses from these groups, and keep the results as they are.

Incidentally, what would be the case if some of the p-values were significant? Thank you all.

  • $\begingroup$ I hope I have answered your questions about Kruskal-Wallis tests. The paragraph about 'weights' and proportional representation could use more detail. What specific questions have given rise to these issues? I have highlighted this paragraph, so that others may be prompted to comment--I hope based on additional details you may include as you edit that part. $\endgroup$ – BruceET Aug 25 at 9:24

A Kruskal-Wallis test in R does not require sample sizes to be the same in all groups. That said, if you have resources to use $kn$ subjects in $k$ groups, power is generally greater if each group has nearly $n$ subjects.

Here are fake data according the the sample sizes you used. They are simulated so that Likert scores for City A tend to be greater than scores for City C, with City B intermediate. The relative proportions of Likert scores 1 through 5 are given by vectors p (R turns these vectors into probability distributions).

a = sample(1:5, 250, rep=T, p = c(1,1,2,3,4))
b = sample(1:5, 125, rep=T, p = c(1,1,3,3,3))
c = sample(1:5, 125, rep=T, p = c(1,2,2,3,3))

table(a); table(b); table(c)
 1  2  3  4  5 
25 17 46 68 94 
 1  2  3  4  5 
11 15 29 38 32 
 1  2  3  4  5 
17 19 33 33 23 

A Kruskal-Wallis test rejects, at the 5% level, the null hypothesis that scores tend to be the same in each of the three groups.


        Kruskal-Wallis rank sum test

data:  list(a, b, c)
Kruskal-Wallis chi-squared = 17.697, 
   df = 2, p-value = 0.0001436

Ad hoc 2-sample Wilcoxon signed-rank tests making pairwise comparisons between cities, show significant differences in scores between Cities A and C. Using the Bonferroni method to guard 'false discovery', one should declare significant differences only for Wilcoxon P-values below about 0.017. So, for my fake data, it would be risky to declare that Cities A and B differ. There is no significant difference between Cities B and C.

[1] 0.03922562
[1] 0.05193156
[1] 4.034746e-05

Smaller sample sizes in Cities B and C make it more difficult to find significant differences in ad hoc comparisons involving those cities. (That is because of the size of the samples, not because of any differences in sizes of cities.)

Assuming that there are several thousand potential subjects in each city, I see no reason why a valid Kruskal-Wallis test would require sample sizes in proportion to city size.

Note: You don't provide much detail about the cluster analysis and principal component analysis. So I will not comment on 'weighting' or proportional sample sizes for those procedures.

  • $\begingroup$ Bruce, the Kruskal-Wallis is clear. Let me to give you some more context. I am analysing people’s opinions to Likert questions on aviation/environment in three cities: Tokyo, Osaka and Fukuoka. Flights go Tokyo-Osaka and back, and Tokyo-Fukuoka and back, for which I chose twice as many respondents from Tokyo than the other two. Similarly, I chose respondents from three age groups 18-34, 35-64 and 65+ in the same proportion 1:2:1 (125, 250, 125 responses, respectively). These are not actual proportions of neither population not age in these cities, therefore I am being advise to “weight” it. $\endgroup$ – holandgents506 Aug 26 at 1:21
  • $\begingroup$ I ran a cluster analysis to divide respondents based in their responses into 3-clusters (based on dendrogram), and then look at their socio-demographic characteristics to see, for example, if people from a similar age, or a similar social background think of act alike. Similarly for PCA, from a set of statements, I look at which ones have the highest influence in people’s answers. Because I already have conducted the survey, I am not sure how to “correct” for the actual population and age distribution. $\endgroup$ – holandgents506 Aug 26 at 1:28
  • $\begingroup$ You might start with some informal exploration. You say you haven't found differences btw cities. If you look at people in some socio-demographic groups, do they tend to have different scores than those in other groups. // After you have a better feel for that, you might ask a question directed precisely at this issue. You might mention you are using K-W to look for differences. But leave out the extraneous stuff about different sample sizes for K-W. Don't try to address too many issues in one question. Focus on one issue per question. $\endgroup$ – BruceET Aug 26 at 3:29
  • $\begingroup$ Hi Bruce, I based the cluster analysis on answers to 6 Likert-questions. As you might remember, I was criticised because the sample obtained from the 3 areas where I applied the questionnaire were not proportional to the actual population. By applying K-W, and obtaining non-significant results (which I did for all 6 questions), I want to argue that there is no statistically significant difference in answers to the questionnaire based on where respondents live, and therefore it should not matter whether the sample is proportional to the population or not. Is this interpretation correct? $\endgroup$ – holandgents506 Sep 9 at 7:00
  • $\begingroup$ When the KW test is run for independent questions the results are all non-significant, but when run for the sum total of Likert answers, like you did in your example above, the p-value is highly significant. $\endgroup$ – holandgents506 Sep 10 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.