I have 2 categorical variables for each observation in my dataset: environment and behaviour. I'm trying to test association between them i.e. does the environment affect behaviour? The resulting contingency table is:

        Behv1    Behv2  Behv3   Behv4
Env 1   54        15      16       0
Env 2   739       201     13       39

I am not sure what statistical test I should use? Normally for association I should use chi-square but some of the cell values are too small, so I think I should Fisher's exact test or is any other that suits best?

Also, I need to report the confidence intervals resulting from the test. How should I do this?


  • $\begingroup$ Have a look at stats.stackexchange.com/questions/367427/… $\endgroup$ – user2974951 Oct 4 '18 at 13:26
  • $\begingroup$ So, I have run the test in R: > fisher.test(mytable) Fisher's Exact Test for Count Data data: mytable p-value = 5.114e-11 alternative hypothesis: two.sided Got the p-value which indicates association. How can I get the confidence levels? $\endgroup$ – Adrian Oct 4 '18 at 13:38
  • $\begingroup$ The fisher.test function already returns a confidence interval (for $\alpha=0.05$ by default). $\endgroup$ – user2974951 Oct 4 '18 at 13:40
  • $\begingroup$ is confidence level same as confidence intervals? $\endgroup$ – Adrian Oct 4 '18 at 13:53
  • $\begingroup$ Confidence level is $1-\alpha$, the fisher.test function returns a CI only for a 2x2 table. $\endgroup$ – user2974951 Oct 4 '18 at 14:02

Out of curiosity I ran the chi-square test (in Stata, but any statistical environment should be up to the task!):

. tabchii 54 15 16 0 \ 739 201 13 39, p

          observed frequency
          expected frequency
          Pearson residual

          |                col                
      row |       1        2        3        4
        1 |      54       15       16        0
          |  62.586   17.047    2.289    3.078
          |  -1.085   -0.496    9.063   -1.754
        2 |     739      201       13       39
          | 730.414  198.953   26.711   35.922
          |   0.318    0.145   -2.653    0.514

2 cells with expected frequency < 5

         Pearson chi2(3) =  94.0651   Pr = 0.000
likelihood-ratio chi2(3) =  51.5291   Pr = 0.000

. ret li

                  r(N) =  1077
                  r(r) =  2
                  r(c) =  4
               r(chi2) =  94.06510751973497
                  r(p) =  2.93238684628e-20
            r(chi2_lr) =  51.52908203554749
               r(p_lr) =  3.77366847284e-11

I note that

  1. Any problem with low expected frequencies is relatively slight. Some texts oversell a old rule-of-thumb that you should worry even if expected frequencies drop below 5, but my experience matches a rule-of-thumb (to be found in Harold Jeffreys, Theory of Probability Oxford University Press, 1961, among other places) that below 1 is the only common danger zone. Here no expected frequency is that low.

  2. Here two flavours of chi-square statistic are not close, which shows some sensitivity, but the choice between P-values of the order of $10^{-20}$ and $10^{-11}$ is scientifically no choice at all. (The exact test confirms overwhelming significance.)

  3. In addition to testing the hypothesis -- the answer from any test is an overwhelming Yes! There is an effect! -- the more interesting question would seem to be what you can learn from the data. The Pearson residuals (observed $-$ expected) / root of expected flag up some fine structure for behaviours 3 and 4.

| cite | improve this answer | |
  • $\begingroup$ Yes, I am trying to learn more about the data. I was told by a reviewer that I should report the confidence levels to understand which categories contributes to the overall differences given by the test. Could you please explain point 3 a little bit more? $\endgroup$ – Adrian Oct 4 '18 at 17:12
  • 1
    $\begingroup$ I agree, I think, with @mdewey. You seem confused between confidence levels (which researchers choose) and confidence intervals (which they calculate). More crucially, sorry, but I can't begin to guess at what the reviewer expects you to do and (although this may seem impolite) I have to worry on your behalf about how competent they are. As an author, reviewer and journal editor, I advise that it's in order to report back that the recommendation is unclear -- or even to ask for more detail. It's a lousy journal that won't allow that. $\endgroup$ – Nick Cox Oct 4 '18 at 17:38
  • 1
    $\begingroup$ The use of residuals from chi-square is, or should be, standard in any text on categorical analysis, but Pearson residuals quantify departure in each cell from expectation and very roughly are expected to have mean and SD about 0 and 1. Haberman in Biometrics 1973 is a basic paper, IIRC. $\endgroup$ – Nick Cox Oct 4 '18 at 17:38

Fisher's exact test gives you the probability of a result as extreme as the one which you observed, for some meaning of extreme. It does not provide an estimate of a statistic about which you could construct a confidence interval. Neither does the $\chi^2$ test for that matter. If you really want to do that you need to decide on a measure of association but it is not clear to me what the scientific question would be which could be answered by a confidence interval for Cramer's $V$ for instance.

| cite | improve this answer | |
  • $\begingroup$ Basically, I need to know which categories are contributing to the overall test of differences because I can't assume that all the categories are different. $\endgroup$ – Adrian Oct 4 '18 at 15:46
  • 2
    $\begingroup$ Which categories are contributing to the test -- all are. Otherwise, see some flavour of residuals, as in my answer. $\endgroup$ – Nick Cox Oct 4 '18 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.