I have been asked to comment on the use of NNT (number needed to treat) and NNH (number needed to harm) in observational studies. My intuition doesn't give me any reasons these would be problematic and a brief Google search did not find any, either.
Of course, there are the usual problems of observational studies, but is there anything particular about NNT and NNH?
I think the problems always arise when NNT and NNH are abused to infer efficacy of treatments. Afterall, statistics doesn't kill people, it's the practioners who do :).
See "Misleading Statistics" (2010, Hutton) (unfortunately, paywalled):
The major claim for NNT was that it is easy to understand, but this claim has been refuted by observational and experimental studies.
For example:
An important problem with NNT relates to its supposed main advantage: it is claimed that it provides a measure in terms of patients treated rather than in terms of probabilities. However, this is not true. An NNT gives the average number of patients, among whom, if they were treated with one therapy rather than another, exactly one patient will benefit. The NNT statistic is biased, and reliable confidence intervals cannot be provided. Furthermore, there is no simple value that indicates no difference between treatments. For meta-analysis, NNT cannot be used directly because simple arithmetic, such as addition, does not give correct results on the NNT scale. If the baseline rate, e.g. the mortality rate on standard treatment, is given, some transformations are required to be able to find the mortality rate on a new treatment using the NNT. If the risk reduction, the difference in rates, is given, the rate on new treatment is found by subtraction.
From "The Numbers Needed to Treat and Harm (NNT, NNH) Statistics: What They Tell Us and What They Do Not" - Andrade, the whole section on Limitations of the NNT is particularly pertinent:
Consider a situation in which drug versus placebo response rates are 12% versus 1%, respectively; the advantage for the drug is 11%, and the NNT is 9. Consider another situation in which the drug versus placebo response rates are 99% versus 88%, respectively; the NNT is again 9. These 2 situations are strikingly different. In the first situation, there is almost no placebo response, and medication is associated with a relatively large treatment gain. In the second situation, there is a large placebo response, and medication is associated with a relatively small treatment gain. Yet, the NNT is the same in the 2 situations. So, it is really important for clinicians to know not only what the unique contribution of the drug is (NNT) but also what the placebo response and nonresponse rates are.
