Quantifying the overlap in sort task results with cells < 5

Question

We have data from two sort tasks where the same items (N=87) were assigned to different types of group ('g' & 'd'). We want to compare the overlap between the assignments to the 'g' groups (N=13) to those of the 'd' groups (N=16). To give you an idea of the data I have created a truncated frequency table (apologies for the table format - not sure how best to show it on here):

In this example 9 items placed into group 'g4' were also placed into 'd1'.

  | g1 | g2 | g3 | g4 | g5 |
----------------------------
d1|  1 |  0 |  0 |  9 |  0 |
----------------------------
d2|  0 |  0 |  0 |  0 |  4 |
----------------------------
d3|  0 |  7 |  4 |  0 |  2 |
----------------------------
d4|  0 |  0 |  8 |  0 |  4 |
----------------------------

The size of the 'g' groups differ and so the potential number of matching items assigned to the 'd' groups depends on the size of the 'g' group. For instance, g4 actually has 18 items assigned to it but only 9 match with d1, g3 has 3 items assigned and none match d1. So the frequencies alone are not particularly representative.

We wanted to have 1) a measure of the overlap between all of the 'g' groups with all of the 'd' groups taking into account the possible overlap, and 2). a global measure of the association between 'g' groupings and 'd' groupings.

I am assuming (please correct me if I am wrong) we can't use Chi-Squared as most of the cells are less than 5 and we can't use Fisher's exact test as it isn't a 2x2 table.

My questions:

1) Are there any other test we can perform to quantify the relationships between these assignments?

2) When I was playing with the data and ran a chi-squared test in R it gave me Pearson's Residual values for each GxD combination - are these residual values OK to use independently of the Chi-Square test? --They seem to represent what we need for our first requirement, i.e. providing an index of the overlap between groups taking into account group size - but I don't want to use these incorrectly.

3). I saw the option of computing p-values by Monte Carlo Simulation in the chisq.test function in R. Could this simulation be used to overcome the problem of cells < 5 with the data?

Answer 1

It's not the observed numbers that need to be anything. It's the expecteds that need to be 'not too small' in order for the chi-square approximation to be reasonable for the usual Pearson chi-square statistic. In any case the rule is very old (over 5 decades) and numerous studies since have argued that it's too conservative.

However, in your case, that issue is in fact a problem because the expected values are quite small.

1) You can use the Pearson test statistic you just have to deal with the actual discrete distribution.

2) You can use an exact test, you just need some way of defining precisely how you want to measure the similarity of assignment/'overlap'.

Are the categories in both sets unordered?

The Pearson residuals aren't independent of the chi-square test; I'm not quite sure what you're asking. They roughly measure (in each cell) how far from what you expect if the two variables were independent, and can be conceived somewhat like a Z-score for that.

3). I saw the option of computing p-values by Monte Carlo Simulation in the chisq.test function in R. Could this simulation be used to overcome the problem of cells < 5 with the data?

Yes, simulation can do exactly that; you basically simulate other tables under whatever is being conditioned on (the null applying and presumably the values in the margins) and compute the test statistic each time, so you can see where your table lies in that distribution.

I don't know if R's built in simulation will work on your table or not, but even if it can't, a custom simulation shouldn't be too hard. The difference between this simulation and an exact test for some measure of association is simply that an exact test counts the proportion of such tables at least as extreme as yours, while simulation estimates that same proportion (with increasing accuracy as the simulation size increases).

The same simulation approach can be used with any other measure of association in the table that you like.

---

I should clarify that "the test will work" means that it's possible to get a test with the desired type I error rate (or lower). It doesn't necessarily mean that the test will have good power when there are very small expected values, or that any of the mentioned statistics are the best possible choice for your problem and the particular circumstances (such as the pattern of expected values).