On the 40000 figure
The news are really sensationalist, but the paper is really well founded. Discussions raged for days in my laboratory, all in all a really necessary critique that makes researchers introspect their work. I recommend the reading of the following commentary by Thomas Nichols, one of the authors of the "Cluster Failure: Why fMRI inferences for spatial extent have inflated false-positive rates" paper (sorry for the long quote).
However, there is one number I regret: 40,000. In trying to refer to
the importance of the fMRI discipline, we used an estimate of the
entire fMRI literature as number of studies impinged by our findings.
In our defense, we found problems with cluster size inference in
general (severe for P=0.01 CDT, biased for P=0.001), the dominant
inference method, suggesting the majority of the literature was
affected. The number in the impact statement, however, has been picked
up by popular press and fed a small twitterstorm. Hence, I feel it’s
my duty to make at least a rough estimate of “How many articles does
our work affect?”. I’m not a bibliometrician, and this really a
rough-and-ready exercise, but it hopefully gives a sense of the order
of magnitude of the problem.
The analysis code (in Matlab) is laid out below, but here is the
skinny: Based on some reasonable probabilistic computations, but
perhaps fragile samples of the literature, I estimate about 15,000
papers use cluster size inference with correction for multiple
testing; of these, around 3,500 use a CDT of P=0.01. 3,500 is about 9%
of the entire literature, or perhaps more usefully, 11% of papers
containing original data. (Of course some of these 15,000 or 3,500
might use nonparametric inference, but it’s unfortunately rare for
fMRI—in contrast, it’s the default inference tool for structural
VBM/DTI analyses in FSL).
I frankly thought this number would be higher, but didn’t realise the
large proportion of studies that never used any sort of multiple
testing correction. (Can’t have inflated corrected significances if
you don’t correct!). These calculations suggest 13,000 papers used no
multiple testing correction. Of course some of these may be using
regions of interest or sub-volume analyses, but it’s a scant few (i.e.
clinical trial style outcome) that have absolutely no multiplicity at
all. Our paper isn’t directly about this group, but for publications
that used the folk multiple testing correction, P<0.001 & k>10, our
paper shows this approach has familywise error rates well in excess of
50%.
So, are we saying 3,500 papers are “wrong”? It depends. Our results
suggest CDT P=0.01 results have inflated P-values, but each study must
be examined… if the effects are really strong, it likely doesn’t
matter if the P-values are biased, and the scientific inference will
remain unchanged. But if the effects are really weak, then the results
might indeed be consistent with noise. And, what about those 13,000
papers with no correction, especially common in the earlier
literature? No, they shouldn’t be discarded out of hand either, but a
particularly jaded eye is needed for those works, especially when
comparing them to new references with improved methodological
standards.
He also includes this table at the end:
AFNI BV FSL SPM OTHERS
____ __ ___ ___ ______
>.01 9 5 9 8 4
.01 9 4 44 20 3
.005 24 6 1 48 3
.001 13 20 11 206 5
<.001 2 5 3 16 2
Basically, SPM (Statistical Parametric Mapping, a toolbox for Matlab) is the most widely used tool for fMRI neuroscience studies. If you check the paper you'll see using a CDT of P = 0.001 (the standard) for clusters in SPM gives nearly the expected family-wise error rate.
The authors even filled an errata due to the wording of the paper:
Given the widespread misinterpretation of our paper, Eklund et al.,
Cluster Failure: Why fMRI inferences for spatial extent have inflated
false-positive rates, we filed an errata with the PNAS Editoral
office:
Errata for Eklund et al., Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates.
Eklund, Anders; Nichols, Thomas E; Knutsson, Hans
Two sentences were poorly worded and could easily be misunderstood as overstating our results.
The last sentence of the Significance statement should read: “These results question the validity of a number of fMRI studies and
may have a large impact on the interpretation of weakly significant
neuroimaging results.”
The first sentence after the heading “The future of fMRI” should have read: “Due to lamentable archiving and data-sharing practices it
is unlikely that problematic analyses can be redone.”
These replace the two sentences that mistakenly implied that our work
affected all 40,000 publications (see Bibliometrics of Cluster
Inference for an guestimate of how much of the literature is
potentially affected).
After initially declining the the errata, on the grounds that it was
correcting interpretation and not fact, PNAS have agreed to publish it
as we submitted it above.
On the so called Bug
Some news also mentioned a bug as the cause of the invalidity of the studies. Indeed, one of AFNI tools was undercorrecting inferences, and this was solved after the preprint was posted in arXiv.
Statistical inference used in functional neuroimaging
Functional neuroimaging includes many techniques that aim to measure neuronal activity in the brain (e.g. fMRI, EEG, MEG, NIRS, PET and SPECT). These are based on different contrast mechanisms. fMRI is based on the blood-oxygen level dependent (BOLD) contrast. In task-based fMRI, given a stimulus, the neurons in the brain responsible for the reception of that stimulation start consuming energy and this triggers the haemodynamic response changing the magnetic resonance signal ($\approx 5\%$) in the vicinity of the recruited micro-vascularization.
Using a generalized linear model (GLM) you identify which voxel signal time-series are correlated with the design of the paradigm of your experiment (usually a boolean timeseries convoluted with a canonical haemodynamic response function, but variations exist).
So this GLM given you how much each voxel time-series resembles the task. Now, say you have two groups of individuals: patients and controls usually. Comparing the GLM scores between the groups could be used to show how the condition of the groups modulates their brain "activation" pattern.
Voxel-wise comparison between the groups is doable, but due to the point-spread function inherent to the equipment plus a smoothing preprocessing step it isn't reasonable to expect voxels individually carry all the information. The difference in voxels among groups should be, in fact, spread over neighboring voxels.
So, cluster-wise comparison is performed, i.e. only differences between groups that form into clusters are considered. This cluster extent thresholding is the most popular multiple comparison correction technique in fMRI studies. The problem lies here.
SPM and FSL depend on Gaussian random-field theory (RFT) for
FWE-corrected voxelwise and clusterwise inference. However, RFT
clusterwise inference depends on two additional assumptions. The first
assumption is that the spatial smoothness of the fMRI signal is
constant over the brain, and the second assumption is that the spatial
autocorrelation function has a specific shape (a squared exponential)
(30)
In SPM at least you have to set a nominal FWE rate and also a cluster-defining threshold (CDT). Basically, SPM finds voxels highly correlated to the task and, after thresholding with the CDT, neighboring ones are aggregated into clusters. These clusters sizes are compared to the expected cluster extent from Random Field Theory (RFT) given the FWER set [1].
Random field theory requires the activity map to be smooth, to be a
good lattice approximation to random fields. This is related to the
amount of smoothing that is applied to the volumes. The smoothing also
affects the assumption that the residuals are normally distributed, as
smoothing, by the central limit theorem, will make the data more
Gaussian.
The authors have shown in [1] that the expected cluster sizes from RFT are really small when comparing with cluster extent thresholds obtained from random permutation testing (RPT).
In their most recent paper, resting-state (another modality of fMRI, where participants are instructed to not think in anything in particular) data was used as if people performed a task during image acquisition, and the group comparison was performed voxel- and cluster-wise. The observed false positive error (i.e. when you observe differences in the signal response to a virtual task between groups) rate should be reasonably lower than the expected FWE rate set at $\alpha = 0.05$. Redoing this analysis millions of times on randomly sampled groups with different paradigms showed most observed FWE rates to be higher than acceptable though.
@amoeba raised these two highly pertinent questions in the comments:
(1) The Eklund et al. PNAS paper talks about "nominal 5% level" of all
the tests (see e.g. horizontal black line on Fig 1). However, CDT in
the same figure is varying and can be e.g. 0.01 and 0.001. How does
CDT threshold relate to the nominal type I error rate? I am confused
by that. (2) Have you seen Karl Friston's reply
http://arxiv.org/abs/1606.08199 ? I read it, but I am not quite sure what
they are saying: do I see correctly that they agree with Eklund et al.
but say that this is a "well known" issue?
(1) Good question. I actually reviewed my references, let's see if I can make it clearer now. Cluster-wise inference is based on the extent of clusters that form after a primary threshold (the CDT, which is arbitrary) is applied. In the secondary analysis a threshold on the number of voxels per cluster is applied. This threshold is based on the expected distribution of null cluster extents, which can be estimated from theory (e.g. RFT), and sets a nominal FWER. A good reference is [2].
(2) Thanks for this reference, didn't see it before. Flandin & Friston argue Eklund et al. corroborated RFT inference because they basically showed that respecting its assumptions (regarding CDT and smoothing) the results are unbiased. Under this light, the new results show different practices in the literature tend to bias the inference as it breaks down the assumptions of RFT.
On the multiple comparisons
It's also well known many studies in neuroscience don't correct for multiple comparisons, estimates ranging from 10% to 40% of the literature. But these are not accounted by that claim, everyone knows these papers have fragile validity and possibly huge false positive rates.
On the FWER in excess of 70%
The authors also reported a procedure that produces FWER in excess of 70%. This "folk"-procedure consists in applying the CDT to keep only highly significant clusters and then applying another arbitrarily chosen cluster-extent threshold (in number of voxels). This, sometimes called "set-inference", has weak statistical bases, and possibly generates the least trustworthy results.
Previous reports
The same authors had already reported on problems with the validity of SPM [1] on individual analyses. There are also other cited works in this area.
Curiously, several reports on group- and individual-level analysis based on simulated data concluded the RFT threshold were, in fact, conservative. With recent advances in processing power though RPT can be performed much more easily on real data, showing great discrepancies with RFT.
UPDATE: October 18th, 2017
A commentary on "Cluster Failure" has surfaced last June [3]. There Mueller et al. argue the results presented in Eklund et al might be due to a specific imaging preprocessing technique used in their study. Basically, they resampled the functional images to a higher resolution before smoothing (while probably not done by every researcher, this is a routine procedure in most fMRI analysis software). They also note that Flandin & Friston didn't. I actually got to see Eklund talk at the same month in the Organization for Human Brain Mapping (OHBM) Annual Meeting in Vancouver, but I don't remember any comments on this issue, yet it seems crucial to the question.
[1] Eklund, A., Andersson, M., Josephson, C., Johannesson, M., & Knutsson, H. (2012). Does parametric fMRI analysis with SPM yield valid results?—An empirical study of 1484 rest datasets. NeuroImage, 61(3), 565-578.
[2] Woo, C. W., Krishnan, A., & Wager, T. D. (2014). Cluster-extent based thresholding in fMRI analyses: pitfalls and recommendations. Neuroimage, 91, 412-419.
[3] Mueller, K., Lepsien, J., Möller, H. E., & Lohmann, G. (2017). Commentary: Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Frontiers in Human Neuroscience, 11.
