0
$\begingroup$

There are two populations called Traditional and Experimental, and a rate is calculated for each. Performance for the Experimental group is considered favorable if the rate is EQUAL TO or LOWER THAN the Traditional rate. Can we use this favorable outcome (equal to or lower than) as the alternative hypothesis (HA)?

The reason for this is so that when the confidence level is higher, we are MORE SURE that the outcome is favorable (if evidence supports). In other words, a higher confidence level should intuitively translate to a higher burden of proof to state a favorable result.

Currently,

  • H0: the Experimental rate is equal to or lower than the Traditional rate (a favorable outcome);
  • HA: the Experimental rate is greater than the Traditional rate (not favorable).

With this setup, a LOWER confidence level (higher alpha) actually requires a higher burden of proof to fail to reject H0. For example, if we reject the null at an 80% confidence level, but with the same evidence, would not reject the null at a 99% confidence level, the 80% confidence level actually leads to a "more cautious" decision because we would not say that there is a favorable outcome given modest evidence to the contrary.

The current approach is using scipy.stats.fisher_exact with the "greater" parameter, so that when Experimental has a significantly higher rate, H0 is rejected and the results are considered unfavorable.

$\endgroup$
0
$\begingroup$

If you can do this with a logistic regression, by all means you should.

What you are describing is a test of non-inferiority or a test of equivalence. Under the null, the experimental group has a higher outcome i.e. it is worse than the control. That means in order to reject the null, the effect in the experimental group has to be less than that of the control group plus a margin. So the null hypothesis is $\mathcal{H}_0: \mu_e > \mu_c + \delta$. That "margin" is a subjective choice based on the knowledge of the condition, the precision of the study, and the hypothesized effect size of the treatment.

The Fisher Exact test is not reported in terms of $\mu$'s: it is usually summarized with a $p$-value. R goes one step further and inverts the hypothesis test to provide 95% CIs for the odds ratio as a measure of association. It makes sense, therefore, to use the log odds ratio as the $\mu$ effect for a non-inferiority design.

Logistic regression makes conducting a non-inferiority test very easy to do. It is possible to calculate the sampling distribution of the odds ratio under the null hypothesis of $\mu_e > \mu_c + \delta$ to come up with inference and confidence intervals for a non-inferiority design. I'm not sure if that's been done, it could be an open area of research.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.