1
$\begingroup$

I'm pretty new to statistical analysis, so sorry if this question is a little basic:

I have a $4\times 2$ table which looks like the following (it lists the number of successful/failed runs of four machines and the total number of times each machine was run): $$ \begin{array}{|c|c|c|c|} \hline & Success & Failed & Total \\ \hline Machine\ A & 90 & 51 & 141\\ \hline Machine\ B & 31 & 23 & 54\\ \hline Machine\ C & 6 & 2 & 8\\ \hline Machine\ D & 152 & 75 & 227\\ \hline \end{array} $$

As far as I have understood it so far, I can test whether there are statistically significant differences between the expected and the observed results with the Chi-squared test.

My question would be: Is there any statistcal test I could use that would tell me, for example: "The success rate for Machine A is statistically significant higher than that of the other machines and thus we can say that Machine A had the best performance"?

Or am I allowed to just directly compare the relative frequency of the success rates? So lets say for example: "Machine C performs the best because it has a success rate of $75\%$, which is the best of all four machines"?

I would be really glad for any input or advice on this, thank you!

$\endgroup$
3
  • $\begingroup$ The fist thing you can do is to produce a confidence interval for each rate of success .. and then compare, and see if they overlap or not .. $\endgroup$ Mar 3, 2022 at 13:55
  • $\begingroup$ Thank you for taking the time to answer! I looked a little bit into that and came up with the following confidence intervals using the adjusted wald method: $$ \begin{array}{|c|c|c|c|} \hline & LBound& UBound\\ \hline Machine\ A & .55 & .71\\ \hline Machine\ B & .44& .69\\ \hline Machine\ C & .4& .93\\ \hline Machine\ D & .61& .73\\ \hline \end{array} $$ So they all kind of overlap somehow. Would thus the interpretation be correct that there is no real winner-"Machine" in terms of success rate? $\endgroup$ Mar 3, 2022 at 14:33
  • $\begingroup$ @MrSmith Checking overlap of confidence intervals is incorrect for two reasons: first, it doesn't achieve the desired confidence level; second--in this particular case--the intervals are not independent. The underlying idea does work when properly applied and is called the Tukey HSD test. One can also formulate ANOVA as a linear regression and perform planned post hoc tests of the coefficients using that standard machinery. $\endgroup$
    – whuber
    Mar 3, 2022 at 14:39

1 Answer 1

0
$\begingroup$

The procedure prop.test in R makes a direct comparison of the proportions of successes for the four 'machines'.

prop.test(c(90,31,6,152), c(141,54,8,227))

        4-sample test for equality of proportions 
        without continuity correction

data:  c(90, 31, 6, 152) out of c(141, 54, 8, 227)
X-squared = 2.1824, df = 3, p-value = 0.5354
alternative hypothesis: two.sided
sample estimates:
   prop 1    prop 2    prop 3    prop 4 
0.6382979 0.5740741 0.7500000 0.6696035 

Warning message:
In prop.test(c(90, 31, 6, 152), c(141, 54, 8, 227)) :
  Chi-squared approximation may be incorrect

Because of the large P-value, the preliminary verdict is not to reject the null hypothesis. That is, no differences are found. However, the small counts for Machine 3, make it impossible to find an accurate p-value.

However, this is roughly equivalent to a chi.squared test on the $4\times 2$ contingency table where rows are for machines and columns are for Success/Failure. As implemented in R, it is possible to do a simulation, for which the P-value is more accurate. The final result still does not find statistically significant differences (5% level) among the four sample proportions of successes, on account of the P-value $0.6472 > 0.05 = 5\%.$

TAB = cbind(c(90, 31, 6, 152), c(51, 22, 2, 75))
chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 1.8077, df = NA, p-value = 0.6472

Very roughly speaking this mirrors the result in your comment with overlapping confidence intervals. However--even if these are 95% confidence intervals-- it is difficult to assess quantitatively the significance of multiple overlaps among them. (See comment of @whuber.)

Notes: Without simulation, the chi-squared test gives a P-value about 0.6 (along with a warning message, not shown).

chisq.test(TAB)$p.val
[1] 0.6132557

The difficulty is low expected values for Machine 3. Because these are the only low expected values, statisticians might differ about the usefulness of the P-value without simulation.

chisq.test(TAB)$exp
           [,1]      [,2]
[1,]  91.699301 49.300699
[2,]  34.468531 18.531469
[3,]   5.202797  2.797203
[4,] 147.629371 79.370629

If we leave Machine 3 out of the analysis altogether, then we do not find any significant differences:

TAB[c(1,2,4),]
     [,1] [,2]
[1,]   90   51
[2,]   31   22
[3,]  152   75  # formerly row 4

chisq.test(TAB[c(1,2,4),])

        Pearson's Chi-squared test

data:  TAB[c(1, 2, 4), ]
X-squared = 1.4481, df = 2, p-value = 0.4848

Finally, if you wanted to compare Machine A with the other three machines taken together, you could do a chi-squared analysis of a $2\times 2$ table with rows for A and BCD and columns for Success/Fail. You should get no warning messages about low counts from that test.

However, if you are going to do several such ad hoc tests on variants of the same data, you should work at level 1% instead of 5% in order to avoid 'false discovery'. (See the Bonferroni method for details.)

$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much for taking the time to answer so thoroughly, actually learned quite a bit from it! $\endgroup$ Mar 4, 2022 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.