2
$\begingroup$

Daryl Bem published a paper called "Feeling The Future" (chronicled in Wired) in which he gives statistical evidence for extrasensory perception (ESP) using traditional statistical techniques in social psychology. This is controversial: is there ESP, or is stats in social psychology broken? Several authors have rebutted his claims, for example saying he did the statistics incorrectly.

I'm trying to understand one statistical claim in the paper.

He ran an experiment with 100 subjects. Each subject had 36 trials (12 neutral, 12 positive, 12 negative), in random order. Each trial was to "pre-identify" which (randomly chosen) curtain hid a picture. In the results, he says:

Across all 100 sessions, participants correctly identified the future position of the erotic pictures significantly more frequently than the 50% hit rate expected by chance: 53.1%, t(99) = 2.51, p = .01, d = 0.25.

In a footnote it says:

I set 100 as the minimum number of participants/sessions for each of the experiments reported in this article because most effect sizes (d) reported in the psi literature range between 0.2 and 0.3. If d = 0.25 and N = 100, the power to detect an effect significant at .05 by a one-tailed, one-sample t test is .80.

In another it says:

Unless otherwise indicated, all significance levels reported in this article are based on one-tailed tests and d is used as the index of effect size.

Which statistical test was he using? What is d? How exactly did he calculate "hit rate" across all users across all trials to be 53.1%?

My understanding is that a t-test would compare two averages. In this case is it an average of averages (average hit rate for a user, then average the user averages), or something else?

$\endgroup$
  • 1
    $\begingroup$ It looks like he computed the percent correct for each subject and then did a t test comparing the mean percent to 50%. $\endgroup$ – David Lane May 25 '17 at 16:31
  • 1
    $\begingroup$ Extraordinary claims require extraordinary evidence. Based on the results quoted here, it's not worth anyone's time to read further. $\endgroup$ – whuber May 25 '17 at 22:03
  • 1
    $\begingroup$ @whuber I think that was Bem's point: using standard, accepted statistical techniques in psychology produces a result we don't believe. So likely there is something wrong with the standard of evaluation in psychology. $\endgroup$ – dfrankow Feb 10 at 20:52
1
$\begingroup$

t-tests generally compare two means, but there is such a thing called a "one-sample t-test," where someone takes one mean and compares it to just any old constant number. In this case, Bem coded a miss as 0 and a hit as 1. In this case, the mean is the same thing as the percent correct.

From there, he could do a one-sample t-test comparing that mean to .50, representing what you would get by guessing by chance.

The d refers to Cohen's d, which is the mean difference divided by the pooled standard deviation. It represents how many standard deviations two means are from one another. In this case, the d represents the mean minus .50 (the constant he compares to), divided by the standard deviation of the scores he measured. This assesses how many standard deviations away from 50% his average hit rate was.

$\endgroup$
  • $\begingroup$ Thanks! I was going to ask if it was a mean of all of the questions (3,600) or a mean of means, but thinking about it, they are probably mathematically exactly the same. And I haven't thought, but maybe the pooled standard deviation is across all people as well? $\endgroup$ – dfrankow May 26 '17 at 20:06
  • $\begingroup$ Since it is a one-sample t-test, I assume the standard deviation for observed scores (each person has a score from 0.00 to 1.00) are used. There's not two standard deviations here, so there is nothing to pool. $\endgroup$ – Mark White May 26 '17 at 20:28
  • $\begingroup$ I was responding to the "pooled standard deviation" in your answer. So if there is nothing to pool, maybe Cohen's d means something slightly different? $\endgroup$ – dfrankow May 27 '17 at 21:29
  • $\begingroup$ Sorry for the confusion--I was speaking about Cohen's d generally. When t-tests compare two different means, the d uses the pooled standard deviation as the denominator. When the t-test involves one sample, the denominator is just the standard deviation of scores. $\endgroup$ – Mark White May 27 '17 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.