Chi-square Test with Specification of Null-Hypothesis Interval

Question

I have a large sample (N > 1.000.000) of a population. Each individual has a treatment associated with it and a class label. There are five actual treatments T1..T5 and there is a reference treatment T0. The class label ranges from C=1..7.

I would like to study if the class label percentages of the reference treatment group is significantly different from any of the real treatment groups. To do that, I am doing five chi-square tests (T0 vs T1, T0 vs T2, ..., T0 vs T5). Due to the large sample size, almost all of my p-values are highly significant, which basically makes them meaningless.

I found a paper from 2018 [1], which introduced the concept of second generation p-values. The idea is that you have an interval around your parameter that you would like to test AND around your null hypothesis. The second generation p-value quantifies how much these two intervals overlap.

I would to know if there is something similar for the chi-square test? In my particular case, I don't care if group percentages are within a range of 0-5% different. I only care about large percentage difference, say, >5% or >10%. Does anybody know of a test that let's me specify this kind of "Don't-Care"-Interval?

[1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0188299#sec001

Answer 1

Normal approximation allows you to compute a confidence interval for the difference between two proportions (this is done for example by the R-function prop.test). You could reject the null hypothesis that the difference between two proportions is at most 5% (say) if everything in the confidence interval is bigger.

In fact, prop.test will also allow you to run a one-sided test of the difference between proportions being a given value such as 5% directly (and hence give out a p-value) specifying the p and alternative parameters. If in fact you're interested in a two-sided test (and therefore in differences +5% and -5%), you will need to piece your result together from two one-sided tests of different p rather than using the two-sided option of the function (that would run a two-sided test of a single difference like +5%, which is not what you want).