6
$\begingroup$

This is a question regarding the binomial test in R.

What happens if I get a p-value < 0.05, which would generally cause me to reject the null hypothesis that the population value is equal to a certain proportion (in this case 0.75), but the hypothesised proportion to be tested (i.e. the null hypothesis) still lies within the 95% confidence interval?

Do I still reject the hypothesis of equal proportions?

Here is an example R code:

x <- 31 
n <- 50
binom.test(x, n, p=0.75)

# p-value = 0.04812
# 95 percent confidence interval:
# 0.4717492 0.7534989
Improve this question
$\endgroup$
2

4 Answers 4

Reset to default
4
$\begingroup$

The problem with tests of binomial proportion is that the tests used are generally approximate (since the exact "Clopper-Pearson" test is ridiculously conservative). Therefore, it's not clear that the procedure used to get the CI is the same as that used to test the hypothesis. Theoretically, either approach should lead to the same conclusion if you are using just one CI and one test.

You have a border case. Either statistic is telling you that your observation is not all that common under the null hypothesis. Remember: there is nothing special about 5% significance...its a cultural artifact from Ronald Fisher in the 1930's. Its a guideline.

For what it's worth, I'd conclude that its unlikely that the true success probability is as high as 0.75.

Per @John

In a strict hypothesis testing situation, you're stuck with your 0.05, so you would reject the null hypothesis under that criteria. However, I wouldn't run to the press quite yet ;-)...hypothesis tests can really destroy any nuance in inference.

Improve this answer
$\endgroup$
8
  • $\begingroup$ The way the questioner is reporting their result they cannot reject 0.75. It's included in the CI and their cutoff is no longer 0.05 in order to maintain alpha of the same. While 0.05 may well be a cultural artifact you don't get to modify it afterwards. The questioner clearly thought it was important going in, ran two tests, and now has ruined both of them. You'd need a lot more information about the question to make a recommendation of reject. $\endgroup$
    – John
    Sep 18, 2015 at 15:04
  • $\begingroup$ @John where did I say reject anything? I was making a general statement about evidence. In either case, 0.75 is an edge case in terms of likelihood. Do with that what you will. My statement about 5% was to help the OP get away from the silly categorical statements that hypothesis tests allow, especially when the formal result is nearly equivocal. $\endgroup$
    – user75138
    Sep 18, 2015 at 15:08
  • $\begingroup$ Your last sentence is reject 0.75. That's what he's testing against. Your goal to help them get away from silly hypothesis test statements is noble but your final sentence is just that. $\endgroup$
    – John
    Sep 18, 2015 at 15:09
  • $\begingroup$ @John I did not say reject I said it appears unlikely. Very different thing. The test results are weak. $\endgroup$
    – user75138
    Sep 18, 2015 at 15:09
  • 1
    $\begingroup$ @John yeah, I was concerned about that after your comment. Their p-value is below their cutoff, so they would reject under that case. $\endgroup$
    – user75138
    Sep 18, 2015 at 15:17
11
$\begingroup$

While the test and confidence interval provided by binom.test() are both exact, the confidence interval is unfortunately not based on inverting the test, so they may lead to inconsistent results. See the paper

Fay, M.P. (2010). Two-sided Exact Tests and Matching Confidence Intervals for Discrete Data. R Journal, volume 2, no. 1, pages 53–58.

for more information. Luckily, there is an R package by the author of the above paper that does provide (three different pairs of) consistent tests and confidence intervals. Applied to your data, the results are [output lightly edited to only show the relevant information]:

> library(exactci)
> binom.exact(31, 50, p=0.75, tsmethod="central")
Exact two-sided binomial test (central method)

p-value = 0.05747
95 percent confidence interval:
0.4717492 0.7534989

> binom.exact(31, 50, p=0.75, tsmethod="minlike")
Exact two-sided binomial test (sum of minimum likelihood method)

p-value = 0.04812
95 percent confidence interval:
0.4799 0.7463

> binom.exact(31, 50, p=0.75, tsmethod="blaker")
Exact two-sided binomial test (Blaker's method)

p-value = 0.04812
95 percent confidence interval:
0.4797 0.7463

Consult the above paper for information on the pros and cons of the three different sets of tests and confidence intervals.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is a nice reference to a well justified package. However, keep in mind that running all of these and picking what you like is called p-hacking. One should be selected for a principled reason. Or perhaps, even better yet, one should just select a method of reporting results that doesn't require one value being under an arbitrary cutoff so much. (BTW, not saying Karl p-hacked. It's not his study and he's not making decisions, just conveying info. But a subsequent reader could take the message that this is what you should do.) $\endgroup$
    – John
    Jan 13 at 16:24
5
$\begingroup$

This is obviously a border case and the CI and test results are not derived in exactly the same manner (the CI is not an inversion of the test). You might want to look up binomial CIs and note that there are many ways they can be calculated all with pluses and minuses. But none of that gets at your central question of whether you should reject the 0.75 hypothesis here.

Furthermore, you're treating your p value and CI's both as tests and, because you did that, you can't reject 0.75 because now you have two tests and you should have done alpha correction on them. The CI should be treated as something else but you've clearly conveyed in your question that you're thinking of it as a test. Given that you can now select whichever test you like alpha is not 0.05.

Step back from testing for a moment.

You need to think about your numbers. For some reason you've selected exactly 0.75 as the amount to reject. What if you had an enormous N and got 0.74 and could reject it on both the CI and test? Would you reach the same conclusion you would reach with the 0.62 you have right now? Is there some range close to 0.75 that's just about equivalent to 0.75 in your work or is exactly 0.75 very critical? If there is a range how much of it is captured by your CI? How believable is your test rejection then? And what about the range of the CI you do have? It's about 0.25 and that's quite a lot of the possible range of proportions. Do you think you think you can say much about what the true proportion really is? It could be some value really close to 0.75 or it could be close to 0.50. How strong a statement do you want to make with the data you have? Also, do values above 0.75 matter the same as values below? Was 0.75 a lower limit you were testing? In this case the conclusion might be different.

So that's a lot of questions but I put them out there to make a point. Simply rejecting the null is going to be a rather pointless endeavour with these data. Assuming you can make a case to reject it, what else can you say? Is it really useful to tell people the true value is not 0.75 but it could be 0.74?

Collect more data.

Improve this answer
$\endgroup$
3
$\begingroup$

This is my first time answering a question, so I'm hoping that I am actually providing a useful answer.

When you run this in R:

x <- 31
n <- 50
p <- 0.75
binom.test(x, n, p = p)

... it returns the following results:

    Exact binomial test

data:  x and n
number of successes = 31, number of trials = 50, p-value = 0.04812
alternative hypothesis: true probability of success is not equal to 0.75
95 percent confidence interval: 
 0.4717492 0.7534989
sample estimates: 
probability of success 
                  0.62

What the p-value of 0.4812 is telling you is that testing for a probability of success of 0.75 (75%) of having 31 "successful" outcomes out of 50 attempts of a binary event (0, 1) -- i.e.: flipping a coin and getting heads up 31 times out of 50 -- is just inside the 95% confidence interval, which are a range of probabilities of success.

Therefore, you could cautiously accept the null hypothesis that the probability of success is equal to 75%. (The alternative hypothesis being that the probability of success is not equal to 75%.)

The calculated probability of success is included at the bottom of the output: 0.62, or 62% chance of success. This is nothing more than 31 / 50.

Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ I would be equally cautious in saying "accepting the null". $\endgroup$
    – chl
    Sep 18, 2015 at 11:38
  • 1
    $\begingroup$ As true as cautiously rejecting it. $\endgroup$
    – Richard Ian Carpenter
    Sep 19, 2015 at 18:42
  • 1
    $\begingroup$ My point was merely about accepting the null in a frequentist framework (strictly speaking we can just reject it), and not how to interpret the present results. In other words, this was just a general remark. $\endgroup$
    – chl
    Sep 19, 2015 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.