I've recently run into an issue where I am trying to test for a statistical difference between two groups, where each element of a group is itself a data object. For any pair of objects, I can calculate a similarity score. I want to check if the groups of objects are statistically different based on those similarity scores.

Assume two groups of objects. Group A ($G_A^{\rightarrow}$) has $N_A$ objects and group B ($G_B^{\rightarrow}$) has $N_B$ objects. Let $x_{i,j}$ be object $j$ from group $i$, where $i$ is in {$A,B$} and $j$ is in {$0,1,...,N_i$} For any pair of objects, I have a function $f(x_{i_1,j_1},x_{i_2,j_2})$ that can give me a scalar score between 0 and 1 for the similarity of any two objects. So then, I can generate the following groups of scores:

  • In-Group Similarities for Group A: $S_A$ = {$f(x_{A,j_1},x_{A,j_2}) \forall j_1,j_2 \in \{0,...,N_A,\}$}
  • In-Group Similarities for Group B: $S_B$ = {$f(x_{B,j_1},x_{B,j_2}) \forall j_1,j_2 \in \{0,...,N_B,\}$}
  • Between-Group Similarities: $S_{A * B}$ = {$f(x_{A,j_1},x_{B,j_2}) \forall j_1 \in \{0,...,N_A,\},j_2 \in \{0,...,N_B,\}$}

I want to know if Group A has a significantly different distribution of objects, using similarity scores as a proxy. It is unclear what the appropriate method to approach this data would be. I either have two samples or three samples, depending on how you look at it. If I have two samples, then one is a combined sample of all in-group similarities (i.e., $S_A \cup S_B$) and the other is the between-group similarities (i.e., $S_{A*B}$). With two samples in hand, I could then hit them with some standard two-sample non-parametric tests.

Alternatively, I could take each of these groups separately and end up with three samples and use tests for more than two groups (e.g., Kruskal-Wallis, Friedman, etc). In either case, it's a nightmare to interpret. It is easier to interpret a comparison of a single in-group sample against the between-group sample (e.g., $S_A$ vs $S_{A*B}$), but then similarity scores within the second group are completely ignored.

Does anyone have experience applying traditional statistical tests to these? I'm trying to stick with them for interpretability. I could probably work out a Bayesian analysis, but that would be less ideal. Overall, there are three primary questions:

  1. Can anyone recommend a good statistical test to apply for this problem?
  2. Would a two sample (in-group vs between group), three sample (all separate), or other group structure be appropriate?
  3. How would one interpret the probabilities generated by analyzing each of these samples?


  • Obviously, I'd exclude duplicate pairings for calculating sets of similarity. I didn't bother representing this because it would be a pain.
  • There are potentially multiple different similarity functions. You can't really rely on what the similarity function will look like, except that it is based on non-linear interactions between the objects (e.g., there is no concept of similarity without two objects).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.