3
$\begingroup$

I have the following question: Are there any statistical methods (e.g. hypothesis tests) that allow you to compare the order of two different lists?

Suppose there are two students : John and Sarah. These two students make a list of the foods they ate in the last week and how often they ate them:

           food frequency name
    ice cream        15 john
 french fries        13 john
         cake        12 john
       oranges        11 john
     pineapple        11 john
         pizza        11 john
         sushi        11 john
       apples        10 john
        celery        10 john
         mango        10 john
         tacos        10 john
        grapes         8 john

           food frequency  name
         sushi        14 sarah
         cake        14 sarah
         pizza        12 sarah
         tacos        12 sarah
        grapes        11 sarah
     pineapple        11 sarah
 french fries        11 sarah
       apples        10 sarah
       oranges        10 sarah
        celery        10 sarah
         mango        10 sarah
    ice cream        10 sarah

Question: Do any statistical methods exist that allow you to determine how similar are the food preferences between both of these students?

For example - looking at this list, I can see that both students ate a lot of cake in the last week (3rd most popular choice for John and 2nd most popular choice of Sarah), and both students did not eat a lot of mango (3rd least popular choice for John and 2nd least popular choice for Sarah).

But apart from these manual comparisons, are there any statistical methods which allow you to compare the similarity between the food preferences between both students? For instance, could some statistical method conclude that the overall preferences of both students are "approximately the same"?

I came across a few methods such as the Wilcox Rank Test (https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test), but I am not sure if this statistical test can be directly applied to this type of question.

Does anyone have any ideas?

Thanks!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Is this a school exercise using some hypothetical situation? Those food choices are very weird and also there seems no real intention to compare the food choices in a meaningful way. When this would have been part of some research then the food choices would be just raw data that first gets converted in some alternative measures before being compared. For instance comparing food composition like caloric intake of different nutrients (fats, proteins, carbohydrates, sugars). Or comparing intake of fruits, grains or other categories. $\endgroup$
    – Sextus Empiricus
    Commented Nov 1, 2021 at 8:05
  • $\begingroup$ @SextusEmpiricus. You're right that this is not the best problem to reinforce whatever statistical principle the author might have had in mind. $\endgroup$
    – BruceET
    Commented Nov 1, 2021 at 17:53
  • $\begingroup$ @sextusempiricus: this is a hypothetical situation i thought of - a friend and I were having a casual discussion about our favorite movies. I was interested in seeing how similar our choices are. Informally, I could tell our movie choices were similar - but would there be a statistical method to quantify how similar our preferences are? $\endgroup$
    – stats_noob
    Commented Nov 1, 2021 at 20:05
  • $\begingroup$ @stats555 it is often better to tell the real underlying motivation. (if there's no clear objections) 1 Lists of favorite movies are a lot different from survey's of food choices. The comparing of lists with food frequencies is a very broad topic and an entire field on it's own. 2 Your question is a bit of an xy problem. You asked for comparing the order of two lists (with identical items), but your underlying problem would not be like comparing order. $\endgroup$
    – Sextus Empiricus
    Commented Nov 1, 2021 at 21:11
  • $\begingroup$ 3 Also confusing is that your problem with the food preferences has frequency as a raw variable and the order is derived. So you got answers that deviated from your situation where the order is to be used, and the answers started explaining ways to use the frequency instead of the order. $\endgroup$
    – Sextus Empiricus
    Commented Nov 1, 2021 at 21:16

2 Answers 2

Reset to default
6
$\begingroup$

Rearranging the data so the foods are in the same order (John's), I got the following vectors, each of 12-counts (you should proofread them):

jo = c(15, 13, 12, 11, 11, 11, 11, 10, 10, 10, 10, 8)
sa = c(15, 13, 12, 11, 11, 11, 11, 10, 10, 10, 12, 11)

Then I plotted the vectors against each other. [First as is, where some points overplot. Then (right panel) with slight jittering (random displacements), so that all 12 foods show as separate points.]

par(mfrow=c(1, 2))
  plot(jo, sa, pch=20)
 Jo = jo + runif(12, -.2, .2) # jittered
 Sa = sa + runif(12, -.2, .2) #
  plot(Jo, Sa)
par(mfrow=c(1, 2))

enter image description here

The plots show a clear positive association (that is, the two students tend to share food preferences).

Spearman's correlation is Pearson correlation of data ranks.

cor(jo, sa, method="spearman")
[1] 0.7078117    # Spearman
cor(rank(jo), rank(sa))
[1] 0.7078117    # Pearson for ranks

A cor.test in R whether the Spearman correlation is significantly different from $0$ does not give an exact P-value because of the ties, but does seem to reject the null hypothesis of $0$ association. (The corresponding test for Pearson correlation assumes that data are normal, which yours are not.)

cor.test(jo, sa, meth="spear")

        Spearman's rank correlation rho

data:  jo and sa
S = 83.566, p-value = 0.01001
alternative hypothesis: true rho is not equal to 0
sample estimates:
      rho 
0.7078117 

Warning message:
In cor.test.default(jo, sa, meth = "spear") :
  Cannot compute exact p-value with ties

Jittering breaks the ties, but loses some information. Even so, a Spearman correlation test on jittered data is still significant at the 5% level. [Moreover, tests on most alternative randomly jittered data also give significant results.]

cor.test(Jo, Sa, meth="s")$p.val
[1] 0.0457531

For more details on the theory and programming of Spearman correlation tests in the presence of ties, see This Q&A.

Notes: (1) I see no reason to do a Wilcoxon test on the paired data. If significant (which it's not), that would show that one student tended to eat more (consistently higher counts) than the other.

wilcox.test(jo, sa, pair=T)$p.val
[1] 0.3710934
Warning message:
In wilcox.test.default(jo, sa, pair = T) :
  cannot compute exact p-value with zeroes

(2) You could use a chi-squared test, but it would be significant if preferences were markedly different. A chi-squared test for homogeneity of counts returns P-value $1.$ The small chi-squared statistic does indicate good agreement between the two students, but correlations are a more direct way to see that.

chisq.test(rbind(jo, sa))

        Pearson's Chi-squared test

data:  rbind(jo, sa)
X-squared = 0.56276, df = 11, p-value = 1
Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ You could also normalize the values; some people eat more frequently than others. Working with a dimensionless "fraction against total" or w/e is also preferred for many methods. I worked hard to make that acronym spell FAT. $\endgroup$
    – Stian
    Commented Nov 1, 2021 at 8:27
  • 3
    $\begingroup$ It seems like the sa vector is quite off right at the start? Sarah eats ice cream 10 times, not 15, to begin with. $\endgroup$
    – Zayn
    Commented Nov 1, 2021 at 15:53
  • $\begingroup$ @Zayn. Sorry if the incorrect vector was a distraction. (Suggested OP proofread; it their hwk.) If you have objections to my method, please elaborate. $\endgroup$
    – BruceET
    Commented Nov 1, 2021 at 17:46
  • 1
    $\begingroup$ @BruceET No, otherwise it's a great answer! I didn't see any issues with the method, but anyone reading this should know not to rely on the conclusions if the input was off. Yes, OP should definitely proofread and do the calculations themselves. $\endgroup$
    – Zayn
    Commented Nov 1, 2021 at 18:41
  • $\begingroup$ @BruceET looks interesting but why is the data for the second variable different from how it should be according to the OP? Could you maybe edit the answer to make it followable? $\endgroup$
    – pd441
    Commented Aug 13 at 11:56
2
$\begingroup$

I would recommend using the discrete form of KL-Divergence to quantify the difference between the distributions. I am assuming that you don't actually care about the order though. You say you do care about the order, "compare the order," but what you describe for the lists has no order "make a list of the foods they ate in the last week and how often they ate them." So the lists appear to be unordered (that is, pizza 10, salad 5 is the same as salad 5, pizza 10, they both mean 10 pizzas and 5 salads were had in the prior week, but they impose no order on the consumption, as opposed to something like a full list with every food item in order of consumption, like [pizza,salad,pizza,pizza,salad,pizza,pizza,pizza,salad,pizza,pizza,pizza,pizza,salad,salad]). If a food is not on a list, then we can infer it to have a frequency of 0 so that we can match the lists, that is, all foods should be in all lists.

KL-Divergence in a nutshell measures how far apart two distributions are (though I use 'measure' loosely here as KL-Divergence is not a metric because the triangle inequality doesn't hold for it).

If you want to also check how likely it is that two lists come from the same distribution (quite a different question, but in case you're interested), you can use the Kolmogorov-Smirnov test. There is an R package that provides both of these for the discrete case called kldtools, https://cran.r-project.org/web/packages/kldtools/kldtools.pdf I have not tried out this package though. To see more about these tests to see if they are suitable for your problem:
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Basic_example
https://machinelearningmastery.com/divergence-between-probability-distributions/
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

See R packages kldtools, KSgeneral, FNN, and LaplacesDemon for KL-Divergence implementations. Rbase has ks.test in the stats package. https://stat.ethz.ch/R-manual/R-patched/library/stats/html/ks.test.html

#install.packages("kldtools")
library(kldtools)
#I will use the same encoding as BruceET
jo = c(15,13,12,11,11,11,11,10,10,10,10, 8)
sa = c(15,13,12,11,11,11,11,10,10,10,12,11)
#Swapping the first and last element, 
#we will use this to show the order 
#of elements doesn't matter.  
jo2 = c(8,13,12,11,11,11,11,10,10,10,10, 15)
sa2 = c(11,13,12,11,11,11,11,10,10,10,12,15)

kldtools(jo,sa)

$KLD
[1] 0.004066538
$KLD.sd
[1] 0.1221033
$KLD.ci.left
[1] -0.01676343
$KLD.ci.right
[1] 0.02489651
$KLD.s
[1] 0.004213292
$KLD.s.sd
[1] 0.1288235
$KLD.s.ci.left
[1] -0.01776308
$KLD.s.ci.right
[1] 0.02618967
$Turing
[1] -0.07779667
$Turing.sd
[1] 0.1221033
$Turing.left
[1] 0.0569667
$Turing.right
[1] 0.09862664

#In the above output, note that this package gives the result of 3 
variations of KL-D, the first is the normal discrete case, the 
second is a symmetric version, and I didn't look into the 3rd one 
much... sorry.  For each version of KL-D it gives 1) the value, 
2) the standard deviation, 3) the left bound of the confidence 
interval, 4) the right bound of the confidence interval.  

kldtools(jo2,sa2)

$KLD
[1] 0.004066538
$KLD.sd
[1] 0.1221033
$KLD.ci.left
[1] -0.01676343
$KLD.ci.right
[1] 0.02489651
$KLD.s
[1] 0.004213292
$KLD.s.sd
[1] 0.1288235
$KLD.s.ci.left
[1] -0.01776308
$KLD.s.ci.right
[1] 0.02618967
$Turing
[1] -0.07779667
$Turing.sd
[1] 0.1221033
$Turing.left
[1] 0.0569667
$Turing.right
[1] 0.09862664

#In the above output, note that it is identical to the first KL-D 
because the order here does NOT matter.  If your description was 
inaccurate and you truly are looking for something where order 
matters, then BruceET's answer is better than this one.  

kldtools(sa,jo)

$KLD
[1] 0.004360046
$KLD.sd
[1] 0.1381221
$KLD.ci.left
[1] -0.01876865
$KLD.ci.right
[1] 0.02748874
$KLD.s
[1] 0.004213292
$KLD.s.sd
[1] 0.1312406
$KLD.s.ci.left
[1] -0.01776308
$KLD.s.ci.right
[1] 0.02618967
$Turing
[1] -0.07782117
$Turing.sd
[1] 0.1381221
$Turing.left
[1] 0.05469247    
$Turing.right
[1] 0.1009499

#For this output, I wanted to demonstrate that while the list order 
doesn't matter, KL-D is not a symmetric test, so giving (sa,jo) vs. 
(jo,sa) DOES matter.  Note though that the second version of KD from 
this package is a symmetric one, so that value is the same.  

#Ok, so we know the distance between these two weekly food 
distributions is small, but what is the chance that these two lists 
were drawn from the same distribution?  
ksboot(sa, jo, 1000)
$ksboot.pvalue
[1] 0.839
$nboots
[1] 1000
There were 50 or more warnings (use warnings() to see the first 50)

#Warnings are from ties because we have exact matching values for 
many of these.  
#Anyway, there is roughly an 84% chance that these two lists come 
from the same distribution.

Finally, I'm not an expert in these concepts, so take this answer with a grain of salt. I may have some mistakes and if so, hopefully some commenters can point them out and/or offer better answers.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ "what you describe for the lists has no order" The frequency of consumption is a variable that allows a ranking. $\endgroup$
    – Sextus Empiricus
    Commented Nov 1, 2021 at 15:18
  • $\begingroup$ @SextusEmpiricus Absolutely, they certainly could be placed in that order, but it is not implied from the description that they should be. They can be ordered, but I'm not seeing any intrinsic reason to order them for comparison. Again, if they are to be, then I think BruceET already has a pretty good answer. $\endgroup$
    – ttbek
    Commented Nov 1, 2021 at 15:23
  • $\begingroup$ Unfortunately, if BruceET's sa vector was wrong, then your analysis results would be wrong too. But the idea of KL-divergence is good, and that's what came to my mind first as well. $\endgroup$
    – Zayn
    Commented Nov 1, 2021 at 15:55
  • $\begingroup$ @ttbek The question ask specifically/explicitly for comparison in terms of order "Are there any statistical methods (e.g. hypothesis tests) that allow you to compare the order of two different lists?". I agree with you that it might not make sense,and it might be an xy-problem, but it is the question. $\endgroup$
    – Sextus Empiricus
    Commented Nov 1, 2021 at 16:28
  • $\begingroup$ @SextusEmpiricus To be frank, I don't care. Hopefully the answer is useful to someone, maybe it won't be, but I see little point in litigating over it. $\endgroup$
    – ttbek
    Commented Nov 1, 2021 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.