I am trying to compare Likert-scale answers to a survey applied in 3 different cities: A, B and C. I arbitrarily chose twice as many respondents from A, for a total of 500 answered questionnaires in the following proportion: A 250 B 125 C 125
Nevertheless, these quantities are not in exact proportion with the actual populations of those cities, for which I am being asked to review my research. I am using a Kruskal-Wallis test to compare answers among the 3 groups, and for all 33 questions in the survey, the p-value is not significant.
Would I be right to argue then that although the sample is not representative of the underlying population, there does not appear to be a difference in responses from the 3 cities?
I have conducted cluster analysis and principal component analysis with these data, and I am being asked to include "weights" to fix the representation issue. My goal is not to have to do this and justify that there is no statistically significant difference in responses from these groups, and keep the results as they are.
Incidentally, what would be the case if some of the p-values were significant? Thank you all.