Independent t-test for between 2 samples, using function other than Mean

Question

I have 2 sample groups of n = 1000 individuals for each group. Each individual rates the quality of a product on a scale from 1 to 10, and these rating are the entire dataset I am working with. I am simply comparing the ratings between groups.

First, I begin by using a simple independent t-test to assess whether there is a statistically significant difference between the means in these two unrelated sample groups.

Second, for these particular product ratings, I am also interested in the difference between groups for another function (other than mean), this function being the:

f(data) = number of times rated as 10 - number of times rated as 6 or lower

Assume that in the first sample, there were 275 ratings of 10, and 171 ratings of 6 or lower, and in the second sample, there were 205 10 ratings of 10, and 118 ratings of 6 or lower. Given these values, the results of this new function on each sample is 275 - 171 = 104 for sample 1, and 205 - 118 = 87 for sample 2.

My question is simply then: is there a statistical test I can use, to be able to confirm with statistical significance that these function values are different between the 2 samples? I assume I cannot simply use another independent t-test, using the function values and the sample variances of the product ratings, although I am not quite sure.

Let me know if I can clarify any parts of the question. Any thoughts on this would be greatly appreciated, thanks!

Answer 1

Using Likert scores, lots of people pretend to have interval data and test these scores using t tests. Using 'number of 10's minus number of below-7's leaves you with only one number per group of 1000.

Two ideas:

(1) Chi-squared homogeneity test. You might admit that the data are fundamentally categorical, make a contingency table with two rows A & B for groups and three columns for <7, 7-9 and 10. Then do a chi-squared test to see if A and B are homogeneous with respect to the three sore categories. If so, look at important differences as guided by large Pearson residuals.

TBL = rbind(c(171,554,275), c(118,677,205))
TBL
     [,1] [,2] [,3]
[1,]  171  554  275
[2,]  118  677  205

out = chisq.test(TBL); out

        Pearson's Chi-squared test

data:  TBL
X-squared = 32.218, df = 2, p-value = 1.009e-07
out$resi

          [,1]      [,2]     [,3]
[1,]  2.204509 -2.478912  2.25924
[2,] -2.204509  2.478912 -2.25924

Comparing observed and expected counts, it seems that group A was more decisive than group B, with substantially fewer 7-9 scores.

An ad hoc comparison: Because of the high emphasis many consumer satisfaction experts put on 'top' Likert scores from surveys, it might be worthwhile seeing if the proportion of 10's differs significantly between the two groups.

prop.test(c(275,205), c(1000,1000), alte="greater")

        2-sample test for equality of proportions 
        with continuity correction

data:  c(275, 205) out of c(1000, 1000)
X-squared = 13.051, df = 1, p-value = 0.0001516
alternative hypothesis: greater
95 percent confidence interval:
 0.0376894 1.0000000
sample estimates:
prop 1 prop 2 
 0.275  0.205 

The sample proportion $0.275$ of 10s in Group A is highly significantly greater than the sample proportion of 10s in Group B.

(2) Two-sample tests on transformed scores. You might transform each individual's score from Likert 1-6 to $-1$ for <7, $0$ for 7-9, and $1$ for 10. Then since you have 1000 in each group, you might get something useful from a 2-sample t test on the transformed data.

a = rep(c(-1,0,1), c(171,554,275))
b = rep(c(-1,0,1), c(118,677,205))
summary(a); sd(a)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -1.000   0.000   0.000   0.104   1.000   1.000 
[1] 0.6600149  # SD a
summary(b); sd(b)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -1.000   0.000   0.000   0.087   0.000   1.000 
[1] 0.5619135  # SD b

Notice that my transformation to values $-1, 0, 1$ has given scores to each person that make it possible to to do a t test on individual subjects.

sum(a)
[1] 104  # 275 "10"s minus 118 "below 7"s
sum(b)
[1] 87   # 205 "10"s minus 118 "below 7"s 

Now we're ready to do a Welch 2-sample t test.

t.test(a,b)

        Welch Two Sample t-test

 data:  a and b
 t = 0.62019, df = 1948.4, p-value = 0.5352
 alternative hypothesis: 
    true difference in means is not equal to 0
 95 percent confidence interval:
  -0.03675809  0.07075809
 sample estimates:
 mean of x mean of y 
     0.104     0.087 

There is no hint that the sample means of the two groups are significantly different.

The data in a and b are highly discrete and so hardly normal, but with a thousand of each there is little doubt that the sample means used in the t test are very nearly normal. So I don't doubt the validity of the outcome: failing to reject the null hypothesis.

Traditionally, Mann-Whitney-Wilcoxon tests have not worked well with data that have many ties, but the implementation of this test in R uses approximations for large samples and does not give an error message about ties. This test fails to find a difference in the 'locations' of the transformed data in a and b. The non-significant P-value of this test is shown below.

wilcox.test(a,b)$p.val
[1] 0.3807537

In summary, I would not say that the idea to focus on scores of 10 and below 7 is a total failure. The chi-squared test gives a highly significant result and it may be worthwhile doing some additional ad hoc tests comparing various proportions.

However, because your t test with the original Likert data gave a highly significant result and because a t test with the transformed data does not even reach near significance, I think it is fair to say that the transformation has resulted in the loss of some potentially important information.