7
$\begingroup$

Over at Skeptics.StackExchange, an answer cites a study into electro-magnetic hypersensitivity:

I am dubious about some of the statistics used, and would appreciate some expertise in double-checking that they are used appropriately.

Figure 5a shows the results of a subject attempting to detect when an electromagnetic field generator was turned on.

Here is a simplified version:

 Actual:   Yes  No
Detected:
  Yes       32  19
  No       261 274

They claim to have used a chi-squared test, and found significance (p < 0.05, without stating what p is.)

The frequencies of the somatic and behavioral responses in the presence and absence of the field were evaluated using the chi-square test (2 × 2 tables) or the Freeman–Halton extension of the Fisher exact probability test (2 × 3 tables; Freeman & Halton, 1951).

I see several problems.

  • They excluded some of the data - see Table 5b - where they left the device off for long periods. I cannot see the justification in separating that data.

  • They seem to be claiming the result is statistically significant when the actual device was on, but not when it wasn't. (I may be misreading this; it isn't clear.) That's not a result that the chi-squared test can give, is it?

  • When I have tried to reproduce this test with an on-line calculator I have found it to be statistically insignificant.

This is my real question: Am I right in saying this?: A two-tailed, chi-squared test using Fisher's Exact Test is the right way to analyze this data AND it is NOT statistically significant.

Improve this question
$\endgroup$
5
  • $\begingroup$ Are the "detected" and "actual" variables observed at the same unit? If so then I'd say this is a symmetry problem. $\endgroup$
    – Momo
    Oct 8, 2012 at 11:00
  • $\begingroup$ @Momo: I think the answer is Yes. There were 600 trials. In 300, the device was actually on, and in the other 300 the device was actually off. The subject was asked if she could detect the electromagnetic radiation, and answered Yes or No. In 14 cases, she failed to answer and they have been excluded. In 535 cases she said No, which means her sensitivity was low (if it existed at all). Not sure how that makes a symmetry problem - any links I could read to learn more? $\endgroup$
    – Oddthinking
    Oct 8, 2012 at 11:09
  • $\begingroup$ Ok, thanks. I just realized that symmetry problem seems to be an expression that is not used in English, so sorry for that. What I mean by it is that the cross classifications are not derived from independent units but that the same unit was repeatedly asked and it therefore is a paired or repeated measurement set up. $\endgroup$
    – Momo
    Oct 8, 2012 at 11:16
  • 2
    $\begingroup$ For the record: There was a Letter to the Editor regarding this paper. It challenged some of the (post-hoc?) classifications of severity used in Table 3a (Experiment 1 and 2), warned of publication bias risks and the need to replicate. It didn't complain about the data in Table 5. $\endgroup$
    – Oddthinking
    Oct 8, 2012 at 11:38
  • 2
    $\begingroup$ It might also be worth noting that this table is right on the margin of appearing "significant": had only a single detection been misclassified, the Fisher test (which is the appropriate one to use) would return a p-value of 10.9%. If the claim is extraordinary or controversial, one would require much stronger evidence than this to accept a conclusion of positive association. $\endgroup$
    – whuber ♦
    Oct 8, 2012 at 15:43

2 Answers 2

Reset to default
3
$\begingroup$

It seems to me that there are three things wrong with the conclusion.

First, as @caracal said: They are reporting "significance" using a one-tailed test, without saying that they are doing so. Most people, I think, recommend using two-tailed tests almost always. Certainly it is not ok to use a one-tail test without saying so.

Second, the effect is tiny. When there was a signal, the subject (there was only one) detected it 11% of the time (32/293). When there was no signal, she detected a signal 6.5% of the time. That difference seems pretty small. And the subject was not able to detect the signal 89% of the time!

Third, as @oddthinking pointed out, there were some selective data reporting that were not properly explained or justified (I didn't read the paper carefully, so am simply repeating what was in the original post).

Improve this answer
$\endgroup$
0
$\begingroup$

A Fisher exact test on the given table gives, per this code

actual <- c(rep("Y", 32), rep("N", 19), rep("Y", 261), rep("N", 274))
det <- c(rep("Y", 51), rep("N", 535))
table(det,actual) 
fisher.test(det,actual)

a p = 0.08

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ would you say that a Fisher test is appropriate for this contingency table? $\endgroup$
    – Momo
    Oct 8, 2012 at 11:24
  • 2
    $\begingroup$ But that's the two-sided p-value. I guess the hypothesis was one-sided ($p(\text{"yes"} | \text{yes}) > p(\text{"yes"} | \text{no})$, giving a p-value of 0.039. $\endgroup$
    – caracal
    Oct 8, 2012 at 12:16
  • $\begingroup$ @caracal: Do you want to elaborate your reasoning and turn this into an answer? $\endgroup$
    – Oddthinking
    Oct 8, 2012 at 12:20
  • $\begingroup$ @Oddthinking Sorry, I currently don't have the time to skim the paper and think about the issues of sampling / experimental design relevant to the question. $\endgroup$
    – caracal
    Oct 8, 2012 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.