Background: We've created an "in-house index" to help quantify something that was never measured/quantified before to aid better communication and provide more objective validations of internal efforts to "change the index" over time.
Question to Answer: Index value in 2018: $I = 0.5$ (say) 2019: $I = 0.6$ Was this change statistically significant or was could it be to random chance?
Null Hypothesis: Nothing interesting happened i.e., the result could be explained through dumb luck alone and not necessarily the outcome of our efforts.
Statistically Significant: Implies that it's possible something we did caused the change. Now we can spend time/effort to figure this out in more detail.
I have never done index based tests of statistical significance and even lack any education in this regard. Here are the questions I'm looking to answer:
- Is there a set of algorithms/procedures for tests of significance for such "index" statistics? Any pointers/suggestions/links?
- Is the "procedure" independent of the index or does one need to constantly rethink everytime you conjure a new index?
- Is index based comparison meaningful and commonly practiced?
- Given that an index may not be based on the "mean" but a computed/derived value how does one go about statistical tests for something like this?
Concrete example: Here are some results I found for "Shannon diversity index" but even they have two different ways:
The Shannon/Simpson index are good examples and what I'm trying to do is quite similar. I've cast this as a "broad" question to "learn" steps involved when confronted with something like this.