I have referred to similar questions on this topic, but the problem presented then was not quite in line with my research, so I have created a new question.

My research is the following: I have conducted a Likert Scale survey with 500 results in two groups, A and B, and 250 responses for each one. Nevertheless, the results are not representative of the underlying population (which is not 50%-50%), therefore any analysis I try to run (e.g. Clustering, PCA, even Pearson's Chi-Squared independence tests) will not be representative of the actual population.

Is there any way to weight results if, for example, the actual proportion was A (88%) and B (12%).

Thank you.

  • $\begingroup$ What would you hope to say about the whole population? Is the main point to see if there is a difference between A and B subgroups? Do you have a summary of Likert scores in Group A and separate summary for Group B.? $\endgroup$ – BruceET Aug 18 at 21:25
  • $\begingroup$ A and B are users of two different kind of services, for example, people who fly low cost and people who fly full service. I want to see if socio demographic characteristics (age, gender, marital status, income level, etc.), as well as responses to Likert-scale questions on quality, importance of air fare, etc., determine people's use of one or the other service. But because my sample (survey answers) was run on a 50-50 population, and in reality the proportion of users is more around 90-10, I am at a loss about how to weight Likert-scale ordinal answers. $\endgroup$ – holandgents506 Aug 18 at 22:21
  • $\begingroup$ In my Answer at the end of the section on t tests, see related comment. // The median Likert opinion is likely closer to the median Likert opinion of (much larger) Group A. [For my fake data, that would be 4.] There is no straightforward way to find the median of the whole from medians of constituent groups. $\endgroup$ – BruceET Aug 18 at 22:46
  • $\begingroup$ @BruceET I didn't answer your initial questions. Yes, the purpose is to see differences between A and B, and yes, I do have a summary, or rather, the full Likert scores separately for A and B. $\endgroup$ – holandgents506 Aug 23 at 11:29

Fake data for illustration. Suppose you have data from Likert scores (1 worst through 5 best) as follows, where Group A seems to have somewhat higher scores than Group B.

table(a); table(b)
 1  2  3  4  5 
12 38 46 73 81 
 1  2  3  4  5 
20 37 68 73 52 

summary(a); summary(b)
a:  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   3.000   4.000   3.692   5.000   5.000 
b:  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0     3.0     3.5     3.4     4.0     5.0 

Welch 2-sample t test. Because you have moderately large sample sizes (250 in each group) it seems reasonable to do a Welch two-sample t test on the (non-normal) Likert scores. Results from R below show P-value about 0.01 so it seems reasonable to say that the Likert average 3.7 for Group A is significantly greater than average 3.4 for Group B.


        Welch Two Sample t-test

data:  a and b
t = 2.7127, df = 497.97, p-value = 0.006905
alternative hypothesis: true difference in means 
  is not equal to 0
95 percent confidence interval:
  0.0805107 0.5034893
sample estimates:
mean of x mean of y 
    3.692     3.400 

If you want to make an overall observation of the population, you might say that the average Likert value is $.88(3.69) + .12(3.40) = 3.66.$ To the extent that you may believe that Likert values imitate valid numerical assessments, this average may be useful.

Wilcoxon test. Some people believe it is better to use a Wilcoxon rank sum test to compare Likert scores for two groups. (Likert data are not numerical, but they are ordinal.) Again there is a significant difference between (medians of) Groups A and B; the P-value is below 1%.

[1] 0.004490969

Chi-squared test of homogeneity. You could also look at a chi-squared test to see if probabilities of counts in the five Likert categories differ between groups A and B. (Categories are treated as nominal.) The two groups have significantly different probabilities for the various Likert scores. The P-value is below 5%.

DTA = rbind(tabulate(a), tabulate(b))
ab.out = chisq.test(DTA); ab.out

        Pearson's Chi-squared test

data:  DTA
X-squared = 12.582, df = 4, p-value = 0.01351

We review the 'observed' counts used, and make sure that 'expected' counts all exceed 5. Also we note that there are a couple of Pearson residuals, possibly large enough to try to interpret.

     [,1] [,2] [,3] [,4] [,5]
[1,]   12   38   46   73   81
[2,]   20   37   68   73   52
     [,1] [,2] [,3] [,4] [,5]
[1,]   16 37.5   57   73 66.5
[2,]   16 37.5   57   73 66.5
     [,1]        [,2]      [,3] [,4]      [,5]
[1,]   -1  0.08164966 -1.456986    0  1.778104
[2,]    1 -0.08164966  1.456986    0 -1.778104

The residuals with largest absolute values are for Likert score 5, where Group A had more (81) than the expected number under the null hypothesis (66.5), while Group B had fewer than expected. To a lesser degree Group B had more Likert 3's than Group B.

The chi-squared statistic (12.6) is the sum of the squares of the Pearson residuals, so one might say that these differences for Likert scores 3 and 5 made important contributions to this significantly large statistic.

Note: In case it is of interest, my fake data were arbitrarily simulated as follows. Vectors p are proportions for Likert scores; R turns them into probabilities.

a = sample(1:5, 250, rep=T, p = c(1, 2, 3, 4, 4))
b = sample(1:5, 250, rep=T, p = c(1, 2, 4, 4, 3))
  • $\begingroup$ Hi Bruce, Thank you so much, this has been enormously helpful. Therefore, I would be right to say that after running these tests, if I get sound enough results (difference in means), I don't really have to weight the data by their actual proportion before conducting any further analysis, correct? $\endgroup$ – holandgents506 Aug 23 at 11:48
  • $\begingroup$ Not sure exactly why you think 'weighting' is necessary. // (a) If you're trying to see if residents of a small city are taller than residents of a large one, it would be OK to measure hts of 250 randomly chosen subjects from each city and compare with t test. (b) With Likert data: There may be more 5's than 2's, but all of the 5's and all of the 2's are present in 2 vectors of length 250 for each group compared using a t test. Data may not be exactly normal, but t test should be sufficiently robust. Weighting wouldn't be appropriate. $\endgroup$ – BruceET Aug 23 at 15:41
  • $\begingroup$ Hi Bruce, I also think the t.test should be enough to compare both groups, even if they haven't been picked in exact proportion to the underlying population. I am being asked to amend this in my research, though. $\endgroup$ – holandgents506 Aug 24 at 2:33
  • $\begingroup$ By the way, I was able to run the t.test fine, but the wilcox.test(a,b)$p.val is giving me an error message. Same when I try to run the code for the Chi-square, DTA = rbind(tabulate(a), tabulate(b)) ab.out = chisq.test(DTA); ab.out I get an error message when I try to tabulate(a) $\endgroup$ – holandgents506 Aug 24 at 2:49
  • $\begingroup$ There is no reason the two samples for a 2-sample t test must be sampled in proportion to population size. Usually, population sizes are much larger than sample sizes. The t test is most powerful for a given total number of subjects if they are equally split between the two populations, but this is not a requirement. $\endgroup$ – BruceET Aug 24 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.