I have two binomial (success=1 and failure=0) datasets and intend to apply a statistical test to show whether they are statistically equivalence or not. First dataset (x) includes around 164 data points in which 58 of them are "1" and the rest are "0". The second dataset/group (y) consists of 280 data points in which 113 of the are "1" and the rest are "0". I applied the following test in r:

  equivalence::tost(x,y,paired = FALSE, epsilon=0.15,conf.level = 0.95, alpha = 0.05)

And I got the following results:

   data:  x and y
   df = 348.24
   sample estimates:
   mean of x mean of y 
   0.3536585 0.4035714 

   Epsilon: 0.15 
   95 percent two one-sided confidence interval (TOST interval):
   -0.12840509  0.02857931
   Null hypothesis of statistical difference is: rejected 
   TOST p-value: 0.01809221 

Regardless of the epsilon value (which is somehow high! and shows the difference between the mean of two population), I think this function is for another type of data, Not binomial data. Does anyone know, Is this the right function to test the equivalency and if not, is there any other test which is appropriate for Binomial data?

I also applied the following test as @user2974951 suggested:

 prop.test(c(58, 113), c(164, 280),alternative = 't')

And I got these results which I can not interpret:

 data:  c(58, 113) out of c(164, 280)
 X-squared = 0.88749, df = 1, p-value = 0.3462
 alternative hypothesis: two.sided
 95 percent confidence interval:
 -0.1477880  0.0479622
 sample estimates:
  prop 1    prop 2 
0.3536585 0.4035714 
  • 1
    $\begingroup$ You say binomial, but you only have zeros and ones in your data, are you sure it's binomial? anyway you could use prop.test() for binomial, or Fisher test for exact test. $\endgroup$ – user2974951 Sep 25 '18 at 12:30
  • 1
    $\begingroup$ Yes, I meant that 1= success and 0=failure. I have edited the question. $\endgroup$ – nahid khosh Sep 25 '18 at 12:35
  • $\begingroup$ Fisher test for exact test is good for categorical data and the null-hypothesis (H0) in this test is that the two groups are equivalent and we should try to reject the H0, but in equivalency test the H0 is that the two population are statistically different and we should find enough evidence that the difference between two populations are not major or the difference is minor. $\endgroup$ – nahid khosh Sep 25 '18 at 13:01

To do an equivalence test, you need some equivalence margins on some appropriate scale. Then you use some method that gives you a valid confidence interval at the desired level (i.e. to perform a test at level $\alpha$ you need a two-sided level $1-\alpha$ confidence interval) and see whether it lies completely within the equivalence boundaries.

E.g. you might want to work with odds-ratios and might think that anything within a factor of 0.8 to 1.25 (equal delta in both directions on the log-odds-ratio scale) is not a meaningful difference. You could then use logistic regression to get and estimated odds ratio (0.809 for group 1 vs. 2) and to get 95% asymptotic Wald confidence intervals (0.542 to 1.206). Since the confidence interval is not completely within 0.8 to 1.25 (i.e. the lower end falls out of it), you would not have shown equivalence (i.e. you cannot reject the null hypothesis of non-equivalence). With other equivalence margins, you might have equivalence.

Often we can also derive a p-value for the decision, but if all you want is "reject null hypothesis" vs. "null hypothesis not rejected", then all you need is confidence interval. When asymptotics apply, you can use the normal approximation to get such a p-value (essentially looking at what level confidence interval would just overlap with the margin, which is easy to do, if the CI is formed via estimated log-odds ratio +- SE * appropriate percentile of the normal distribution such as 1.96), while with more sparse data this can be a bit difficult.

You could of course also decide that you are not interested in an odds ratio and want a risk difference or a risk ratio.

example <- data.frame(group = c(rep(1,164), rep(2,280)),
                      outcome = c(rep(1,58), rep(0,164-58), rep(1,113), rep(0,280-113)))

glmfit1 <- glm(data = example,
               formula = outcome ~ factor(group),
               family = "binomial")


# Get estimate and standard error
# use negative coefficient because of choice of reference group
estimate <- -summary(glmfit1)$coefficients[2,1] 
se <- summary(glmfit1)$coefficients[2,2]

# Wald confidence interval on the odds-ratio scale, you get a test decision simply 
# by comparing your NI margin to the CI limits (if the CI limits extend
# beyond it, you cannot reject the null hypothesis of non-equivalence).
# Here we are assuming that you want 95% CIs corresponding to a 5% level test
wald_ci <- c(exp( estimate + se* qnorm(0.975) ),
             exp( estimate - se* qnorm(0.975) ))

# If you really need a p-value, then you can do the following:

# Let us assume this is your equivalence margin:
nimargin <- log(2) 
# p-value using large sample normality assumption (corresponding to Wald confidence limits)
pvalue <- min(1, 2*(1-pnorm( (abs(nimargin) - abs(estimate))/se, 0, 1)))

The example code above gives you a p-value for equivalence of 0.0183, if your NI margin is log(2) on the logit-scale. So for that NI margin you could reject the null hypothesis of non-equivalence and conclude that within those limits the groups are equivalent. On the other hand for a NI margin of 1.25, you would get a p-value from the equivalence test of 0.9579, so you could not reject that null hypothesis.

| cite | improve this answer | |
  • $\begingroup$ Yes, the difficult part is to select the margin. As I shown in the above example "equivalence::tost()" I used epsilon=0.15 as the margin between the mean of two population. As the groups are binomial, the mean of two groups are the success rate, e.g., the mean of group1 is 0.35 and for group2 is 0.40, so success rates are the same respectively. So my question is how can I interpret the epsilon value? is it a percentage of difference? if I decrease that value 0.15 to 0.10, the test can not reject the null-hypothesis! $\endgroup$ – nahid khosh Sep 26 '18 at 8:02
  • $\begingroup$ My second question is: Is "equivalence::tost()" test a right test for binomial data? If it is not, is odd-ratio an appropriate test to check whether the groups are equivalent considering that that two groups are binomial? Is there any function in r that can apply the odd ratio? $\endgroup$ – nahid khosh Sep 26 '18 at 8:03
  • 1
    $\begingroup$ The glm function in the stats package would be the standard way of doing a logistic regression (which has as its default effect measure the odds ratio, if you use the default logit-link function). Note: coefficients provided will be on the log-odds-ratio scale (not odds ratio scale). As far as I am aware equivalence::tost is assuming a linear regression model with normally distributed error terms (i.e. t-test), so it would typically not be appropriate for binomial data (asymptotically it might well be fine). $\endgroup$ – Björn Sep 26 '18 at 9:02
  • $\begingroup$ Applying and using glm() is bit a confusing, is possible to guide how I can construct the test in r? $\endgroup$ – nahid khosh Sep 26 '18 at 10:34
  • 1
    $\begingroup$ They are odds ratios between the groups. $\endgroup$ – Björn Oct 13 '18 at 11:34

Interpretation of the test of two proportions. There should be no difficulty interpreting the result of prop.test (suggested by @user2974951): The two sample proportions are $58/164=0.354$ and $113/280=0.404,$ as shown at the end of the output. Also, the P-value of the test is $0.35>0.05$ so your proportions are not significantly different at the 5% level.

Looking at confidence intervals. If you need additional analyses to be satisfied that there is no significant difference, I suggest you look at Agresti-Coull CI's for the two population proportions. [This slight modification of the traditional confidence interval, has been shown to have better coverage probability.]

With $X$ successes in $n$ trials, let $\check p = \frac{X+2}{n+4}.$ Then the 95% CI is of the form $$\check p \pm 1.96\sqrt{\frac{\check p(1-\check p)}{n+4}}.$$ For the first case the CI is $(.285,.430)$ and for the second the CI is $(.348,.462).$ These two CIs overlap massively, illustrating again that there is no significant difference.

Power analysis. In order to get a test that can successfully distinguish between population probabilities $.35$ and $.40$ about $n=2000$ observations on each are required, as the following power analysis from Minitab shows.

enter image description here

So with sample sizes around 200 you're very far from an adequate sample size to distinguish between the two methods; that is, to have probability 0.90 or 0.95 of rejecting $H_0: p_1 = p_2$ against the alternative $H_a: p_1 < p_2$ when $p_1 \le .35$ and $p_2 = .40.$

In public opinion polling to predict an election, where two candidates may each have about half of the vote, one must interview about $2500$ subjects to predict individual candidates' support within $\pm 0.02;$ about $1100$ for $\pm 0.03.$

| cite | improve this answer | |
  • 1
    $\begingroup$ That would be a misinterpretation of a hypothesis test. Failure to reject the null hypothesis of no difference is not in itself evidence for no difference (equivalence). $\endgroup$ – Björn Sep 26 '18 at 6:12
  • $\begingroup$ My point is that this is in response to a question for an equivalence test. $\endgroup$ – Björn Sep 26 '18 at 16:24
  • $\begingroup$ I consistently said "no significant difference." Not sure which part you're objecting to. If you will quote offending/misleading sentence I'll gladly edit it. // My power analysis clearly shows difference btw .35 and .45 would be considered an important difference. And I made it clear I was explaining prop.test--2nd half of question. $\endgroup$ – BruceET Sep 26 '18 at 16:25
  • $\begingroup$ That test is not the right approach for showing that there is no meaningful difference, which was what was asked for in the question. $\endgroup$ – Björn Sep 26 '18 at 16:32
  • $\begingroup$ I am still struggling to apply glm() function for the equivalency test --as @Björn suggested-- for the binomial data that I asked above. I tried with "binomial.test()" function in r, which includes glm() function inside, but it seems is not for the equivalency test. do you have any other suggestion or something that I can go a bit forward. $\endgroup$ – nahid khosh Oct 12 '18 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.