# Weighting in Likert Scale results

I have referred to similar questions on this topic, but the problem presented then was not quite in line with my research, so I have created a new question.

My research is the following: I have conducted a Likert Scale survey with 500 results in two groups, A and B, and 250 responses for each one. Nevertheless, the results are not representative of the underlying population (which is not 50%-50%), therefore any analysis I try to run (e.g. Clustering, PCA, even Pearson's Chi-Squared independence tests) will not be representative of the actual population.

Is there any way to weight results if, for example, the actual proportion was A (88%) and B (12%).

Thank you.

• What would you hope to say about the whole population? Is the main point to see if there is a difference between A and B subgroups? Do you have a summary of Likert scores in Group A and separate summary for Group B.? – BruceET Aug 18 at 21:25
• A and B are users of two different kind of services, for example, people who fly low cost and people who fly full service. I want to see if socio demographic characteristics (age, gender, marital status, income level, etc.), as well as responses to Likert-scale questions on quality, importance of air fare, etc., determine people's use of one or the other service. But because my sample (survey answers) was run on a 50-50 population, and in reality the proportion of users is more around 90-10, I am at a loss about how to weight Likert-scale ordinal answers. – holandgents506 Aug 18 at 22:21
• In my Answer at the end of the section on t tests, see related comment. // The median Likert opinion is likely closer to the median Likert opinion of (much larger) Group A. [For my fake data, that would be 4.] There is no straightforward way to find the median of the whole from medians of constituent groups. – BruceET Aug 18 at 22:46
• @BruceET I didn't answer your initial questions. Yes, the purpose is to see differences between A and B, and yes, I do have a summary, or rather, the full Likert scores separately for A and B. – holandgents506 Aug 23 at 11:29

Fake data for illustration. Suppose you have data from Likert scores (1 worst through 5 best) as follows, where Group A seems to have somewhat higher scores than Group B.

table(a); table(b)
a
1  2  3  4  5
12 38 46 73 81
b
1  2  3  4  5
20 37 68 73 52

summary(a); summary(b)
a:  Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.000   3.000   4.000   3.692   5.000   5.000
b:  Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.0     3.0     3.5     3.4     4.0     5.0


Welch 2-sample t test. Because you have moderately large sample sizes (250 in each group) it seems reasonable to do a Welch two-sample t test on the (non-normal) Likert scores. Results from R below show P-value about 0.01 so it seems reasonable to say that the Likert average 3.7 for Group A is significantly greater than average 3.4 for Group B.

t.test(a,b)

Welch Two Sample t-test

data:  a and b
t = 2.7127, df = 497.97, p-value = 0.006905
alternative hypothesis: true difference in means
is not equal to 0
95 percent confidence interval:
0.0805107 0.5034893
sample estimates:
mean of x mean of y
3.692     3.400


If you want to make an overall observation of the population, you might say that the average Likert value is $$.88(3.69) + .12(3.40) = 3.66.$$ To the extent that you may believe that Likert values imitate valid numerical assessments, this average may be useful.

Wilcoxon test. Some people believe it is better to use a Wilcoxon rank sum test to compare Likert scores for two groups. (Likert data are not numerical, but they are ordinal.) Again there is a significant difference between (medians of) Groups A and B; the P-value is below 1%.

wilcox.test(a,b)$p.val [1] 0.004490969  Chi-squared test of homogeneity. You could also look at a chi-squared test to see if probabilities of counts in the five Likert categories differ between groups A and B. (Categories are treated as nominal.) The two groups have significantly different probabilities for the various Likert scores. The P-value is below 5%. DTA = rbind(tabulate(a), tabulate(b)) ab.out = chisq.test(DTA); ab.out Pearson's Chi-squared test data: DTA X-squared = 12.582, df = 4, p-value = 0.01351  We review the 'observed' counts used, and make sure that 'expected' counts all exceed 5. Also we note that there are a couple of Pearson residuals, possibly large enough to try to interpret. ab.out$$obs [,1] [,2] [,3] [,4] [,5] [1,] 12 38 46 73 81 [2,] 20 37 68 73 52 ab.out$$exp [,1] [,2] [,3] [,4] [,5] [1,] 16 37.5 57 73 66.5 [2,] 16 37.5 57 73 66.5 ab.out$resi
[,1]        [,2]      [,3] [,4]      [,5]
[1,]   -1  0.08164966 -1.456986    0  1.778104
[2,]    1 -0.08164966  1.456986    0 -1.778104


The residuals with largest absolute values are for Likert score 5, where Group A had more (81) than the expected number under the null hypothesis (66.5), while Group B had fewer than expected. To a lesser degree Group B had more Likert 3's than Group B.

The chi-squared statistic (12.6) is the sum of the squares of the Pearson residuals, so one might say that these differences for Likert scores 3 and 5 made important contributions to this significantly large statistic.

Note: In case it is of interest, my fake data were arbitrarily simulated as follows. Vectors p are proportions for Likert scores; R turns them into probabilities.

set.seed(818)
a = sample(1:5, 250, rep=T, p = c(1, 2, 3, 4, 4))
b = sample(1:5, 250, rep=T, p = c(1, 2, 4, 4, 3))

• Hi Bruce, Thank you so much, this has been enormously helpful. Therefore, I would be right to say that after running these tests, if I get sound enough results (difference in means), I don't really have to weight the data by their actual proportion before conducting any further analysis, correct? – holandgents506 Aug 23 at 11:48
• Not sure exactly why you think 'weighting' is necessary. // (a) If you're trying to see if residents of a small city are taller than residents of a large one, it would be OK to measure hts of 250 randomly chosen subjects from each city and compare with t test. (b) With Likert data: There may be more 5's than 2's, but all of the 5's and all of the 2's are present in 2 vectors of length 250 for each group compared using a t test. Data may not be exactly normal, but t test should be sufficiently robust. Weighting wouldn't be appropriate. – BruceET Aug 23 at 15:41
• Hi Bruce, I also think the t.test should be enough to compare both groups, even if they haven't been picked in exact proportion to the underlying population. I am being asked to amend this in my research, though. – holandgents506 Aug 24 at 2:33
• By the way, I was able to run the t.test fine, but the wilcox.test(a,b)\$p.val is giving me an error message. Same when I try to run the code for the Chi-square, DTA = rbind(tabulate(a), tabulate(b)) ab.out = chisq.test(DTA); ab.out I get an error message when I try to tabulate(a) – holandgents506 Aug 24 at 2:49
• There is no reason the two samples for a 2-sample t test must be sampled in proportion to population size. Usually, population sizes are much larger than sample sizes. The t test is most powerful for a given total number of subjects if they are equally split between the two populations, but this is not a requirement. – BruceET Aug 24 at 2:50