Background: We've created an "in-house index" to help quantify something that was never measured/quantified before to aid better communication and provide more objective validations of internal efforts to "change the index" over time.

Question to Answer: Index value in 2018: $I = 0.5$ (say) 2019: $I = 0.6$ Was this change statistically significant or was could it be to random chance?

Null Hypothesis: Nothing interesting happened i.e., the result could be explained through dumb luck alone and not necessarily the outcome of our efforts.

Statistically Significant: Implies that it's possible something we did caused the change. Now we can spend time/effort to figure this out in more detail.

I have never done index based tests of statistical significance and even lack any education in this regard. Here are the questions I'm looking to answer:

  1. Is there a set of algorithms/procedures for tests of significance for such "index" statistics? Any pointers/suggestions/links?
  2. Is the "procedure" independent of the index or does one need to constantly rethink everytime you conjure a new index?
  3. Is index based comparison meaningful and commonly practiced?
  4. Given that an index may not be based on the "mean" but a computed/derived value how does one go about statistical tests for something like this?

Concrete example: Here are some results I found for "Shannon diversity index" but even they have two different ways:

The Shannon/Simpson index are good examples and what I'm trying to do is quite similar. I've cast this as a "broad" question to "learn" steps involved when confronted with something like this.


There are a few issues here that stem from basic but common misunderstandings about statistics. As a result, your hypotheses are not properly formulated, and it is unclear what you are actually trying to test.

There is no such thing as a "statistically significant" difference between two known values: Statistical "significance" is a measure of the magnitude of evidence of a difference between unknown quantities (or for some other hypothesis pertaining to unknown quantities. The "significance" here refers to the magnitude of evidence of a difference, not the magnitude of the difference itself. In the absence of some hypothesis about unknown quantities, there is no statistical hypothesis test at all, and no "statistically significance" arises.

In this particular case you have two known values $I_{2018} = 0.5$ and $I_{2019} = 0.6$. Since $0.5 \neq 0.6$ those values are different, with certainty. There is no legitimate sense in which we would say that the difference between the number $0.5$ and the number $0.6$ is "statistically significant". They are different numbers, with certainty; that is all.

An index is only useful insofar as it predicts real things: Presumably the purpose of your index is to predict something about reality.You say that it was developed "to help quantify something that was never measured/quantified before". Okay, great. So presumably what you actually want to test is either: (1) whether or not your index corresponds to that thing it is measuring (assuming the latter can be accurately measured in some competing way); or (2) whether or not your index has predictive value in predicting things that are affected by that thing it is measuring. Either way, you will need to formulate some hypothesis test with hypotheses that relate your index to other things in reality.

There is no such thing as a "test of significance" for an index, taken in isolation of any clear hypotheses. The only time there is a "significance test" of an index is when you formulate hypotheses that postulate relationships between your index and some other thing in reality. Statistical hypothesis testing occurs when the index is postulated to relate to some unknown quantity, and we obtain statistical evidence about the relationship. As stated above, the hypothesis test determines whether there is "statistically significant" evidence of the postulated relationship.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.