I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same.

There are lots of distance measures that I am aware of (e.g. list of statistical distance measures), but many seem to be tailored to work with continuous distributions. Is there a go to distance measure or test with categorical data? (Literature welcome)

Also, ideally I would want to compute a formal test (say p-values), but I assume I can always run a permutation test?

  • $\begingroup$ Do you want a distance between two probability vectors of the same lenght? Such as, to compare (.10, .25, .25, .20, .20) with (.15, .25, .10, .30, .20)? They are empirical distributions of some 5-category categorical distribution. $\endgroup$
    – ttnphns
    Sep 29 '21 at 23:44
  • $\begingroup$ @ttnphns yes. See my answer to BruceET below $\endgroup$
    – Dan.phi
    Sep 30 '21 at 7:07
  • 1
    $\begingroup$ Geometric and Harmonic mean similarities, Bhattacharyya distance, Hellinger distance, Chi-square distance for probabilities, Pearson/Neyman chi-square divergence (asymmetric and symmetric), Kullback–Leibler asymmetric divergence (Information gain), Kullback–Leibler symmetric divergence (Jeffreys divergence), K-divergence (asymmetric), K-divergence (symmetric, Topsoe distance), Jensen difference (Information radius), Taneja distance. $\endgroup$
    – ttnphns
    Sep 30 '21 at 7:51
  • 1
    $\begingroup$ ... you may find formulas on my web-page (link on my profile), in collection "Various proximities", macro !KO_proxqnt, find in .docx document. $\endgroup$
    – ttnphns
    Sep 30 '21 at 7:54
  • $\begingroup$ Great! My question was then whether any of those is better suited for categorical data than the rest? $\endgroup$
    – Dan.phi
    Sep 30 '21 at 8:02

I assume you have two samples of categorical data and want to know if they were taken from the same distribution. Perhaps most commonly, to compare nominal categorical samples, one can use a chi-squared test.

For the fictitious data below, levels of the categorical variables are denoted by $1$ through $5,$ but those are not ordered labels.

Suppose we have populations with two different probability distributions $p_1$ and $p_2$ as follows:

p1 = c(.1,.1,.1,.2,.5)  # heavy emphasis on last category
p2 = c(.1,.2,.2,.2,.3)  # less heavy

Use R to sample data vectors x1 and x2 (of different sizes) both from population with probabilities $p_1.$

x1 = sample(1:5,1000,rep=T,prob=p1)
t1 = tabulate(x1, nbins=5); t1
[1] 105  99  85 196 515             # tabulation of first sample
x2 = sample(1:5,1500,rep=T,prob=p1)
t2 = tabulate(x2, nbins=5); t2
[1] 157 154 132 289 768             # tabulation of second

Then take a sample 'y` from the population with probabilities $p_2.$

y = sample(1:5,1500,rep=T,prob=p2)
t = tabulate(y, nbins=5); t
[1] 155 287 320 309 429

Use a chi-squared test to compare samples x1 and x2, using the appropriate table TABs of counts:

TBLs = rbind(t1,t2);  TBLs
   [,1] [,2] [,3] [,4] [,5]
t1  105   99   85  196  515
t2  157  154  132  289  768

With P-value above 5% the test finds no difference between these two samples from the same population.


        Pearson's Chi-squared test

data:  TBLs
X-squared = 0.18744, df = 4, p-value = 0.9959

By contrast, with P-value near $0,$ a chi-squared test overwhelmingly rejects the null hypothesis that the two samples were taken from the same population.

TBLd = rbind(t2,t);  TBLd
   [,1] [,2] [,3] [,4] [,5]
t2  157  154  132  289  768
t   155  287  320  309  429


        Pearson's Chi-squared test

data:  TBLd
X-squared = 215, df = 4, p-value < 2.2e-16

The chi-squared statistic $H = \sum_{i=1}^2\sum_{j=1}^5 \frac{(X_{ij} = E_{ij})^2}{E_{ij}}$ (labeled X-squared in output) can be used as a measure of disagreement between the two samples tested. Here, $i=1,2$ sample rows and $j=1,2,3,4,5$ column categories of the table and the $E_{ij}$ are found in the usual way, using row and column totals.

For example, in the second chi-squared test above, the expected counts (assuming the null hypothesis to be true) of the ten cells are shown below:

   [,1]  [,2] [,3] [,4]  [,5]
t2  156 220.5  226  299 598.5
t   156 220.5  226  299 598.5

In particular, $E_{11} = \frac{(1500)(302)}{3000} = 156.0.$ (In this simple example, this amounts to the average of $157$ and $155.)$ If all of the $E_{ij} > 5,$ then $H \sim\mathsf{Chisq}(\nu),$ where $n = (r-1)(c-1),$ for a table with $r$ rows and $c$ columns.

  • $\begingroup$ Thanks! To clarify, rather than the samples, I was interested in actual measures of distance between (empirical) probability distributions. E.g. should I use total variation distance with such a discrete distribution? Or are there results suggesting some particular distance measure works better with categorical data? (things like Kolmogorov–Smirnov don't work great) $\endgroup$
    – Dan.phi
    Sep 29 '21 at 21:53
  • 1
    $\begingroup$ For samples, the chi-squared statistic $H$ is a measure of the disagreement between the two samples. I did not refer to the Wikipedia article; Unless I missed something, it does not seem to say much about about the comparisons you want to make. Perhaps look for 'effect size' here $\endgroup$
    – BruceET
    Sep 29 '21 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.