This paper by Massey and Denton 1988 is a fairly prolific overview of commonly used indices in Sociology/Demography. It would also be useful for some other key terms used for searching articles. Frequently in Sociology the indices are labelled with names such as "hetereogeinity" and "segregation" as well as "diversity".
Part of the reason no absolute right answer exists to your question is that people frequently only use epistemic logic to reason why one index is a preferred measurement. Infrequently are those arguments so strong that one should entirely discount other suggested measures. The work of Massey and Denton is useful to highlight what many of these indices theoretically measure and when they differ to a substantively noticeable extent (in large cities in the US).