2
$\begingroup$

I am comparing two different groups based on working status. For this instance I am looking at occupation so I have a table that looks like so:

mat1 <- matrix(c(90, 32, 9, 8, 46, 7, 8, 5), 
         4,2); dimnames(mat1) <- list(c("A"," B", "C", "D"), 
         c("Working", "Not Working"))
#   Working Not Working
# A      90          46
# B      32           7
# C       9           8
# D       8           5

I initially used the p-value of my chi-square test to determine any significant difference between the working and not working group. It has been suggested to me that I instead describe and compare the magnitudes of the differences with confidence intervals. I know that for a 2x2 table I could use prop.test to get confidence intervals, but that is not the case for the above table. Does anyone know how I would go about obtaining those confidence intervals?

$\endgroup$

1 Answer 1

1
$\begingroup$

First, let's see if there are differences in the proportion working across the four groups A, B, C, D. (Data similar to yours.)

w = c(90, 32, 9, 3)
nw = c(46, 7 , 8, 5)
TBL = rbind(w, nw)
chisq.test(TBL)

        Pearson's Chi-squared test

data:  TBL
X-squared = 8.7062, df = 3, p-value = 0.03346

Warning message:
In chisq.test(TBL) : 
 Chi-squared approximation may be incorrect

The low cell counts in groups C and D, trigger a warning message, putting the validity of the P-value into doubt. The version of 'chisq.test` implemented in R, allows for simulation of a more accurate P-value, showing a significant effect at the 5% level.

chisq.test(TBL, sim=T)$p.val
[1] 0.03098451

Significance barely at the 5% level does not invite extensive ad hoc tests. To avoid false discovery they should show significance at lower levels. Furthermore, it is not clear just which confidence intervals would be of interest. A look at the Pearson residuals to see if there groups that are strikingly different, possibly suggests comparing groups A and B. However, the level of significance there is unimpressive, especially if we protect against false discovery.

chisq.test(TBL)$resi
         [,1]      [,2]       [,3]      [,4]
w  -0.1173306  1.148334 -0.7081676 -1.019365
nw  0.1671828 -1.636247  1.0090588  1.452480

chisq.test(TBL[,c(1,2)], cor=F)

        Pearson's Chi-squared test

data:  TBL[, c(1, 2)]
X-squared = 3.6176, df = 1, p-value = 0.05717

You have already said you know how to use 'prop.test' to get a 95% confidence interval for the difference of proportions in A and B.

I don't see a point in looking at other pairs of groups---especially not, in view of the low counts there. Maybe you would like to compare group A with the other three groups combined, but 'prop.test' can handle that.

If you had additional kinds of analyses in mind using confidence intervals, please be more specific, and maybe one of us can help.

$\endgroup$
6
  • 1
    $\begingroup$ I appreciate the help, however you have input the wrong values for w, it should 90, 32, 9, 8, which results in a p-value of 0.1247 (0.1234 when calculating the more accurate p-value). Upon inspection of the residuals it now appears that B and C are the most different, resulting in a p-value of 0.02371, again nothing extraordinary, but as you mentioned I think I will probably compare A to the other 3 groups combined. $\endgroup$
    – A. Gelfand
    Sep 11, 2021 at 3:35
  • $\begingroup$ Good. My intention was to illustrate the method, and it seems you understand that well. $\endgroup$
    – BruceET
    Sep 11, 2021 at 5:09
  • 3
    $\begingroup$ @A.Gelfand If that was helpful, please consider accepting or upvoting as in stats.stackexchange.com/help/someone-answers $\endgroup$
    – Bernhard
    Sep 11, 2021 at 5:52
  • 1
    $\begingroup$ @BruceET you are putting too much emphasis on hypothesis testing, and the warning about the accuracy of the Pearson $\chi^2$ is overly cautious. The Pearson test can be very accurate down to expected frequencies of 1.0. And note that there are exact solutions for this problem if you switch to Bayes. $\endgroup$ Sep 11, 2021 at 11:49
  • $\begingroup$ @FrankHarrell. Thanks for your comment. I would be curious to see your preferred analysis of either the real data or my slight variation of it. // In this particular case I think it is important to note that my change of one of the counts from 8 to 3 (out of a grand total of 200), changed a P-value of about .12 to about .03, thus putting in doubt the validity of any ad hoc exploration, even among groups where the data are unchanged. It seems one has to be a bit humble about the practical meaning of 'accuracy' when analyzing any such table. $\endgroup$
    – BruceET
    Sep 11, 2021 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.