1
$\begingroup$

An experiment was run 10 times in configuration a and there were A failures, 10 times in configuration b giving B failures, 10 times ... (in all 9 different configurations). I have a list of 9 numbers giving the number of failures for each configuration (out of 10).

4 4 4 3 3 2 0 2 10

I suspect the configuration makes no difference and that the failure counts might have a binomial distribution (it is tempting to classify that 10 as an outlier and remove it).

Feeding the above list of values into R's prop.test function, along with the probabilities for a binomial distribution with p=0.3556 (the mean of failure count/10) generates a warning that the Chi-squared approximation may be incorrect; not surprising giving the small sample size.

A qq-plot would also suffer from the same sample size issues.

Is there a "Fisher's exact test" equivalent for testing whether a sequence of values might be drawn from a binomial distribution?

A related but different experiment produced a failure count sequence of (no outliers here :-):

10 8 7 10 9 9 10 8 7

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ My suspicion is that the 10 is recorded successes, not failures, and is really a 0. Probably not checkable, but it is the obvious explanation. $\endgroup$
    – jbowman
    Jun 18, 2012 at 23:36

2 Answers 2

Reset to default
2
$\begingroup$

Including the ten in the analysis you get a significant result and without the 10 you don't get a significant result from prop.test in R. You can simulate your situation in R in order to see if you trust the approximate result:

# number of simulation replicates
n <- 10000

# variable to hold the output
is10 <- rep(NA, times = n)

for(i in 1:n){
    # 10 experiment runs for 9 configurations
    mySeq <- rbinom(n = 9, size = 10, prob = .3556)
    # count how many times you see 10 failures
    num10 <- sum(mySeq > 9)
    # assign 1 if at least one 10, 0 otherwise
    is10[i] <- if(num10 >= 1){1} else {0}
}

# across all replicates, how many times was 10 seen at least once
sum(is10)

When I ran this I saw 10 failures only once out of 10000 replicates, even when the probability is calculated with your data that include seeing 10 failures one time, so I think it is safe to say that configuration 9 is significantly different from the other configurations. Whether this is due to something about configuration 9 or due to it being an outlier you have to decide.

Improve this answer
$\endgroup$
3
  • $\begingroup$ dbinom(0:10, 10, 0.3556) tells me that 10 failures is very rare. $\endgroup$
    – Derek Jones
    Jun 18, 2012 at 15:36
  • $\begingroup$ If p=0.3556 10 failures out of 10 is rare. But that doesn't really address my question which is why do you expect that the configuration doesn't matter. Using the estimate 0.3556 already presupposes that they are the same by pooling the results. Also if configuration 9 is ignored you would get an estimate of 0.275 which is what you would use if you thought that the 10 out of 10 for configuration 9 is suspect. $\endgroup$
    – Michael R. Chernick
    Jun 18, 2012 at 15:40
  • 2
    $\begingroup$ Michael, sorry for the delay in replying. R's glm function found no connection between configuration and failures, looking at it from a factorial design point of view found nothing useful (I am using somebody else's data and cannot rerun the experiments). My only justification for removing the 10 would involve hand waving and suggesting that perhaps a mistake was made in running a test 90 times (10 times for 9 configurations). $\endgroup$
    – Derek Jones
    Jun 18, 2012 at 15:48
2
$\begingroup$

There is a generalization of the Fisher Exact test for $2 \times2$ Tables to more general $R \times C$. In your case $R=2$ and $C=9$. The null hypothesis is that the column proportion is independent of the row. So it tests a null hypothesis that the proportions are the same for each configuration versus the alternative that they differ. So the test is good for telling you when they differ (if the sample size is large enough to have power to reject when at least one proportion differs by a designated meaningful difference). But with a small sample size it is not likely to be able to reject even for reasonably large differences. The observed value of $10$ in your case might lead to rejecting the null hypothesis. Why would you suspect that something is wrong with configuration $9$? Also you haven't really explained why you suspect that the configurations do not affect the failure rate. What is your reasoning for that?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.