# Does this article use an inappropriate chi-square test or some other test of significance?

I am trying to understand a result from an academic article analysing paragraphs of text. It reports a significance of p < .001, which is obviously strongly significant. Unfortunately, I cannot replicate this calculation. I assume that this is due to my limited knowledge of statistics, but I would like to rule out author error.

In the text, the author reports that any of the features of interest was more likely to occur than no feature of interest: 48 paragraphs with features, 12 features without, p < .001. No further details are given. This is the finding that I am interested in.

There is a table that reports the occurrence of all the features across multiple genres of text. I cannot replicate the calculation for degrees of freedom displayed here and a chi-square test seems inappropriate for such low values. (The variables have been anonymized but the values are unchanged.)

What significance test might have provided the p < .001 finding?
How could the chi-square value for the table have been calculated?
Is there an obvious justification for using a chi-square test with such low values?

• If it's an academic article, why you can't you cite it and why do you need to mangle its data? – Nick Cox Jan 5 '17 at 22:17
• It's confusing because "Some feature" is a subtotal. Eliminating that from the table gives precisely the stated results for the $\chi^2$ and df values. However, none of this is consistent with $p\lt 0.001$. The p-value for this $\chi^2$ statistic is $17.5\%$ (using a Monte-Carlo distribution). If we test the subtotal against "No feature" then $\chi^2=1.875, p=0.55$ is not significant either. Can you supply any additional information about the $p\lt 0.001$ claim? @Nick One might suppose the OP is a reviewer of an article whose details must be protected until it is published. – whuber Jan 5 '17 at 22:17
• I agree on all counts. All puns intended. – Nick Cox Jan 5 '17 at 22:29
• @Nick Cox. Apologies, I should have cited: Weissberg, R.C. (1984) ‘Given and New: Paragraph development models from scientific English’, TESOL Quarterly, 18:3, 485-500. The text is p.492; the table is p.493. I was attempting to provide the relevant information to people who don't have journal access, without infringing copyright, but missed the obvious fact that some people do have access. – EAP Matt Jan 5 '17 at 22:30
• The text "any of the features of interest was more likely to occur than no feature of interest: 48 paragraphs with features, 12 features without," is clearly not a reference to the chi-square n the larger table, but only to the comparison of the proportion with features vs $\frac12$ – Glen_b Jan 6 '17 at 4:54

The report of 8 df implies to me 5 rows by 3 columns in the table used, and a few moments' work reproduces the result, with the easy guess that "Some feature" is just a secondary tabulation not echoed in the chi-square (which would indeed be absurd double counting). I use Stata here, but the calculations should be trivial in your favourite software. If not, change your favourite software.

. tabchii 9 2 8 \ 2 3 2 \ 1 6 2 \ 5 3 5 \ 3 6 3, replace

observed frequency
expected frequency

-------------------------------
|         col
row |     1      2      3
----------+--------------------
1 |     9      2      8
| 6.333  6.333  6.333
|
2 |     2      3      2
| 2.333  2.333  2.333
|
3 |     1      6      2
| 3.000  3.000  3.000
|
4 |     5      3      5
| 4.333  4.333  4.333
|
5 |     3      6      3
| 4.000  4.000  4.000
-------------------------------

12 cells with expected frequency < 5

Pearson chi2(8) =  11.5941   Pr = 0.170
likelihood-ratio chi2(8) =  12.2947   Pr = 0.139


Those worried by the large number of observed frequencies below 5 should be cheered by none being below 1 and by strong consistency with a Fisher's exact test.

. tab row col [w=observed], exact
(frequency weights assumed)

Enumerating sample-space combinations:
stage 5:  enumerations = 1
stage 4:  enumerations = 7
stage 3:  enumerations = 142
stage 2:  enumerations = 887
stage 1:  enumerations = 0

|               col
row |         1          2          3 |     Total
-----------+---------------------------------+----------
1 |         9          2          8 |        19
2 |         2          3          2 |         7
3 |         1          6          2 |         9
4 |         5          3          5 |        13
5 |         3          6          3 |        12
-----------+---------------------------------+----------
Total |        20         20         20 |        60

Fisher's exact =                 0.167


Most careful accounts of the problem of small observed frequencies emphasise that the real danger zone is when they go below 1. An often overlooked example is Harold Jeffreys, Theory of Probability.

• Thank you for explaining how the chi-square is calculated in the table, for offering reassurance on the small numbers, and for raising the issue of Fisher's exact test. I now suspect that Weissberg ran a Fisher's exact test on a 1x2 table (48 and 12). This indeed produces a p < .001 because p = 0.00*. I have never seen a 1x2 table used in this way and I am concerned that this would make the test invalid, but I cannot find a clear statement either way. – EAP Matt Jan 5 '17 at 23:22
• @EAPMatt: How did you perform Fisher's Exact test on the 1x2 table? It's not obvious what that means. – Scortchi - Reinstate Monica Jan 6 '17 at 12:01
• It's pleasing to have my answer accepted and upvoted, but it was an answer to the original version of the question. @Scortchi's comment now seems to be the core point, and my answer is focused on a trivial uncertainty at best. – Nick Cox Jan 6 '17 at 12:12
• "it should indicate the probability of obtaining the same, or a more extreme, values given a constant marginal frequency" - the question's which marginal frequency. Fisher's Exact Test as usually understood is concerned with equal but unspecified marginal frequencies of two proportions. At any rate the web tool seems to have plumped for one-half & to be returning standard binomial test results. – Scortchi - Reinstate Monica Jan 6 '17 at 17:54
• @EAPMatt: "Constant probability of success" enters explanations of binomial trials to contrast with the changing probability of success over the course of trials involving sampling from finite populations. I can't really see why you think it's a problem here, though I haven't looked at the paper. And I can't say the significance test is misguided, just that it strikes me as possibly unnecessary. – Scortchi - Reinstate Monica Jan 6 '17 at 19:32

The phrasing

any of the features of interest was more likely to occur than no feature of interest: 48 paragraphs with features, 12 features without, p < .001

suggests to me a test of the null hypothesis that the probability of a paragraph's having some feature of interest is one-half, using the binomial distribution. Why that null hypothesis should be of particular interest is another question.

What significance test might have provided the p < .001 finding?

Nick Cox has provided information about analysis of the table you provide using the $\chi^2$ test and an exact test. Neither resulted in p<0.001.

The results are presented in the paper as follows:

The only way I could think that the authors of the paper arrived at this p-value is to apply a test of proportion on the marginal result that 48/60 showed some features and 12/60 showed no features. See below, from Stata

. prtesti 60 48 60 12, count

Two-sample test of proportions                     x: Number of obs =       60
y: Number of obs =       60
------------------------------------------------------------------------------
Variable |       Mean   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
x |         .8   .0516398                      .6987879    .9012121
y |         .2   .0516398                      .0987879    .3012121
-------------+----------------------------------------------------------------
diff |         .6   .0730297                      .4568645    .7431355
|  under Ho:   .0912871     6.57   0.000
------------------------------------------------------------------------------
diff = prop(x) - prop(y)                                  z =   6.5727
Ho: diff = 0

Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
Pr(Z < z) = 1.0000         Pr(|Z| > |z|) = 0.0000          Pr(Z > z) = 0.0000


This is erroneous, of course, as pointed out by @whuber in the comments below.

Nevertheless, it is one way to explain the results they've reported.

• 48+12=60 demonstrates you are testing one half of a dataset against the other half! This is invalid due to the perfect negative correlation between the two halves. – whuber Jan 5 '17 at 22:35
• I agree. I'm not saying that it's correct. I'm providing a possible explanation for the results that the OP found. – user140401 Jan 5 '17 at 22:37
• I see--you might want to emphasize that point of view in your post. To me, this explanation looks too implausible. Although there are good reasons to suppose many published papers have statistical errors, this one looks just too elementary and misguided to be a serious consideration. – whuber Jan 5 '17 at 22:39
• That's a good point. I'll make the edit. – user140401 Jan 5 '17 at 22:40
• I still don't understand how this can construed as a two-sample test of proportions. There appears to be only one sample. The quotation you found suggests that perhaps the proportion of "no feature" among the sample of 60 is being compared to $1/2$--but that's a one-sample test. – whuber Jan 5 '17 at 22:59