I've recently run into an issue where I am trying to test for a statistical difference between two groups, where each element of a group is itself a data object. For any pair of objects, I can calculate a similarity score. I want to check if the groups of objects are statistically different based on those similarity scores.
Assume two groups of objects. Group A ($G_A^{\rightarrow}$) has $N_A$ objects and group B ($G_B^{\rightarrow}$) has $N_B$ objects. Let $x_{i,j}$ be object $j$ from group $i$, where $i$ is in {$A,B$} and $j$ is in {$0,1,...,N_i$} For any pair of objects, I have a function $f(x_{i_1,j_1},x_{i_2,j_2})$ that can give me a scalar score between 0 and 1 for the similarity of any two objects. So then, I can generate the following groups of scores:
- In-Group Similarities for Group A: $S_A$ = {$f(x_{A,j_1},x_{A,j_2}) \forall j_1,j_2 \in \{0,...,N_A,\}$}
- In-Group Similarities for Group B: $S_B$ = {$f(x_{B,j_1},x_{B,j_2}) \forall j_1,j_2 \in \{0,...,N_B,\}$}
- Between-Group Similarities: $S_{A * B}$ = {$f(x_{A,j_1},x_{B,j_2}) \forall j_1 \in \{0,...,N_A,\},j_2 \in \{0,...,N_B,\}$}
I want to know if Group A has a significantly different distribution of objects, using similarity scores as a proxy. It is unclear what the appropriate method to approach this data would be. I either have two samples or three samples, depending on how you look at it. If I have two samples, then one is a combined sample of all in-group similarities (i.e., $S_A \cup S_B$) and the other is the between-group similarities (i.e., $S_{A*B}$). With two samples in hand, I could then hit them with some standard two-sample non-parametric tests.
Alternatively, I could take each of these groups separately and end up with three samples and use tests for more than two groups (e.g., Kruskal-Wallis, Friedman, etc). In either case, it's a nightmare to interpret. It is easier to interpret a comparison of a single in-group sample against the between-group sample (e.g., $S_A$ vs $S_{A*B}$), but then similarity scores within the second group are completely ignored.
Does anyone have experience applying traditional statistical tests to these? I'm trying to stick with them for interpretability. I could probably work out a Bayesian analysis, but that would be less ideal. Overall, there are three primary questions:
- Can anyone recommend a good statistical test to apply for this problem?
- Would a two sample (in-group vs between group), three sample (all separate), or other group structure be appropriate?
- How would one interpret the probabilities generated by analyzing each of these samples?
Notes:
- Obviously, I'd exclude duplicate pairings for calculating sets of similarity. I didn't bother representing this because it would be a pain.
- There are potentially multiple different similarity functions. You can't really rely on what the similarity function will look like, except that it is based on non-linear interactions between the objects (e.g., there is no concept of similarity without two objects).
