1
$\begingroup$

Current problem: We have a batch of $n$ items for which we capture their details with $m$ attributes. It could look something like this:

Matrix representation

The goal is to compute an "index" that says how "similar" this batch is (or how "diverse" it is ($1-similarity$?))

I've looked at things like the Simpson's index but that requires things to be grouped as "species" and I don't have such a thing in my domain. Even the Jaccard index doesn't seem to cater to my need since it compares 2 populations and I don't have such discrete sets. I'm not sure if entropy indexes would help but am open to suggestions and would like to see some examples of problem/computation.

What's are some "indexes" that I could explore prior to rolling my own (enjoyably pursuing actively but nothing concrete/satisfying yet)? Any references/articles/pointers to examples that are close to what I'm trying to achieve and could "map" my problem to that?

Requirement: If every item is the "same" across all $m$ attributes, then $diversity = 0$ (similarity is $1$) else if all are different then $diversity = 1$ (or similarity is $0$) and other combinations somewhere in between.

The problem could be equivalently reformulated as "Each entity has a particular DNA string representation - how similar is the group?", if that helps.

UPDATE: After mulling further I was wondering if I could use the Simpson's index by auto-collating "species" in my data set - rows with the "same values" across dimensions are grouped together and maybe assumed as a species. The count of the number of "organisms" in a group $n$ and total number of such groups $N$ could feed into the formula below:

$$D = 1-\frac{\sum n(n-1)}{N(N-1)}$$

Would this work?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.