There are two populations called Traditional and Experimental, and a rate is calculated for each. Performance for the Experimental group is considered favorable if the rate is EQUAL TO or LOWER THAN the Traditional rate. Can we use this favorable outcome (equal to or lower than) as the alternative hypothesis (HA)?
The reason for this is so that when the confidence level is higher, we are MORE SURE that the outcome is favorable (if evidence supports). In other words, a higher confidence level should intuitively translate to a higher burden of proof to state a favorable result.
- H0: the Experimental rate is equal to or lower than the Traditional rate (a favorable outcome);
- HA: the Experimental rate is greater than the Traditional rate (not favorable).
With this setup, a LOWER confidence level (higher alpha) actually requires a higher burden of proof to fail to reject H0. For example, if we reject the null at an 80% confidence level, but with the same evidence, would not reject the null at a 99% confidence level, the 80% confidence level actually leads to a "more cautious" decision because we would not say that there is a favorable outcome given modest evidence to the contrary.
The current approach is using scipy.stats.fisher_exact with the "greater" parameter, so that when Experimental has a significantly higher rate, H0 is rejected and the results are considered unfavorable.