Current problem: We have a batch of $n$ items for which we capture their details with $m$ attributes. It could look something like this:

Matrix representation

The goal is to compute an "index" that says how "similar" this batch is (or how "diverse" it is ($1-similarity$?))

I've looked at things like the Simpson's index but that requires things to be grouped as "species" and I don't have such a thing in my domain. Even the Jaccard index doesn't seem to cater to my need since it compares 2 populations and I don't have such discrete sets. I'm not sure if entropy indexes would help but am open to suggestions and would like to see some examples of problem/computation.

What's are some "indexes" that I could explore prior to rolling my own (enjoyably pursuing actively but nothing concrete/satisfying yet)? Any references/articles/pointers to examples that are close to what I'm trying to achieve and could "map" my problem to that?

Requirement: If every item is the "same" across all $m$ attributes, then $diversity = 0$ (similarity is $1$) else if all are different then $diversity = 1$ (or similarity is $0$) and other combinations somewhere in between.

The problem could be equivalently reformulated as "Each entity has a particular DNA string representation - how similar is the group?", if that helps.

UPDATE: After mulling further I was wondering if I could use the Simpson's index by auto-collating "species" in my data set - rows with the "same values" across dimensions are grouped together and maybe assumed as a species. The count of the number of "organisms" in a group $n$ and total number of such groups $N$ could feed into the formula below:

$$D = 1-\frac{\sum n(n-1)}{N(N-1)}$$

Would this work?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.