# 40,000 neuroscience papers might be wrong

I saw this article in the Economist about a seemingly devastating paper [1] casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical assumptions." I read the paper and see it's partly a problem with multiple comparison corrections, but I'm not an fMRI expert and am finding it difficult to follow.

What are the erroneous assumptions the authors are talking about? Why are those assumptions made? What are ways around making these assumptions?

A back of the envelope calculation says 40,000 fMRI papers is over a $billion in funding (grad student salary, operating costs, etc.). [1] Eklund et al., Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates, PNAS 2016 • See also the paper about the fMRI of dead salmon. wired.com/2009/09/fmrisalmon – Sycorax Jul 25 '16 at 17:20 • It's a really fine line, specially in the study of neuropathologies, because you can totally restrict false positives as much as you want, but in the tradeoff you end with huge false negative incidence. Jul 25 '16 at 17:50 • There's a bit of a difference between the two: the salmon paper is a nice little parable about the importance of multiple comparisons correction, which everyone should have been doing already. In contrast, the PNAS issue bites people who were trying to do "the right thing", but the correction itself was a bit wonky. Jul 25 '16 at 17:50 • I think this is a great question to ask because it gets to the heart of multiple comparisons corrections and key assumptions in conducting this type of analysis in the context of a common vein of research. However, the only question in the text is "does someone with more knowledge than em want to comment on it?" which is somewhat broad and nonspecific. If this could be focused to a particular statistical issue within the scope of the help center, it would be more suited to this forum. – Sycorax Jul 25 '16 at 19:00 • Thanks. I edited the question to make it more specific. Let me know if I should edit it more. Jul 25 '16 at 19:17 ## 1 Answer ## On the 40000 figure The news are really sensationalist, but the paper is really well founded. Discussions raged for days in my laboratory, all in all a really necessary critique that makes researchers introspect their work. I recommend the reading of the following commentary by Thomas Nichols, one of the authors of the "Cluster Failure: Why fMRI inferences for spatial extent have inflated false-positive rates" paper (sorry for the long quote). However, there is one number I regret: 40,000. In trying to refer to the importance of the fMRI discipline, we used an estimate of the entire fMRI literature as number of studies impinged by our findings. In our defense, we found problems with cluster size inference in general (severe for P=0.01 CDT, biased for P=0.001), the dominant inference method, suggesting the majority of the literature was affected. The number in the impact statement, however, has been picked up by popular press and fed a small twitterstorm. Hence, I feel it’s my duty to make at least a rough estimate of “How many articles does our work affect?”. I’m not a bibliometrician, and this really a rough-and-ready exercise, but it hopefully gives a sense of the order of magnitude of the problem. The analysis code (in Matlab) is laid out below, but here is the skinny: Based on some reasonable probabilistic computations, but perhaps fragile samples of the literature, I estimate about 15,000 papers use cluster size inference with correction for multiple testing; of these, around 3,500 use a CDT of P=0.01. 3,500 is about 9% of the entire literature, or perhaps more usefully, 11% of papers containing original data. (Of course some of these 15,000 or 3,500 might use nonparametric inference, but it’s unfortunately rare for fMRI—in contrast, it’s the default inference tool for structural VBM/DTI analyses in FSL). I frankly thought this number would be higher, but didn’t realise the large proportion of studies that never used any sort of multiple testing correction. (Can’t have inflated corrected significances if you don’t correct!). These calculations suggest 13,000 papers used no multiple testing correction. Of course some of these may be using regions of interest or sub-volume analyses, but it’s a scant few (i.e. clinical trial style outcome) that have absolutely no multiplicity at all. Our paper isn’t directly about this group, but for publications that used the folk multiple testing correction, P<0.001 & k>10, our paper shows this approach has familywise error rates well in excess of 50%. So, are we saying 3,500 papers are “wrong”? It depends. Our results suggest CDT P=0.01 results have inflated P-values, but each study must be examined… if the effects are really strong, it likely doesn’t matter if the P-values are biased, and the scientific inference will remain unchanged. But if the effects are really weak, then the results might indeed be consistent with noise. And, what about those 13,000 papers with no correction, especially common in the earlier literature? No, they shouldn’t be discarded out of hand either, but a particularly jaded eye is needed for those works, especially when comparing them to new references with improved methodological standards. He also includes this table at the end:  AFNI BV FSL SPM OTHERS ____ __ ___ ___ ______ >.01 9 5 9 8 4 .01 9 4 44 20 3 .005 24 6 1 48 3 .001 13 20 11 206 5 <.001 2 5 3 16 2  Basically, SPM (Statistical Parametric Mapping, a toolbox for Matlab) is the most widely used tool for fMRI neuroscience studies. If you check the paper you'll see using a CDT of P = 0.001 (the standard) for clusters in SPM gives nearly the expected family-wise error rate. The authors even filled an errata due to the wording of the paper: Given the widespread misinterpretation of our paper, Eklund et al., Cluster Failure: Why fMRI inferences for spatial extent have inflated false-positive rates, we filed an errata with the PNAS Editoral office: Errata for Eklund et al., Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Eklund, Anders; Nichols, Thomas E; Knutsson, Hans Two sentences were poorly worded and could easily be misunderstood as overstating our results. The last sentence of the Significance statement should read: “These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results.” The first sentence after the heading “The future of fMRI” should have read: “Due to lamentable archiving and data-sharing practices it is unlikely that problematic analyses can be redone.” These replace the two sentences that mistakenly implied that our work affected all 40,000 publications (see Bibliometrics of Cluster Inference for an guestimate of how much of the literature is potentially affected). After initially declining the the errata, on the grounds that it was correcting interpretation and not fact, PNAS have agreed to publish it as we submitted it above. ## On the so called Bug Some news also mentioned a bug as the cause of the invalidity of the studies. Indeed, one of AFNI tools was undercorrecting inferences, and this was solved after the preprint was posted in arXiv. ## Statistical inference used in functional neuroimaging Functional neuroimaging includes many techniques that aim to measure neuronal activity in the brain (e.g. fMRI, EEG, MEG, NIRS, PET and SPECT). These are based on different contrast mechanisms. fMRI is based on the blood-oxygen level dependent (BOLD) contrast. In task-based fMRI, given a stimulus, the neurons in the brain responsible for the reception of that stimulation start consuming energy and this triggers the haemodynamic response changing the magnetic resonance signal ($\approx 5\%$) in the vicinity of the recruited micro-vascularization. Using a generalized linear model (GLM) you identify which voxel signal time-series are correlated with the design of the paradigm of your experiment (usually a boolean timeseries convoluted with a canonical haemodynamic response function, but variations exist). So this GLM given you how much each voxel time-series resembles the task. Now, say you have two groups of individuals: patients and controls usually. Comparing the GLM scores between the groups could be used to show how the condition of the groups modulates their brain "activation" pattern. Voxel-wise comparison between the groups is doable, but due to the point-spread function inherent to the equipment plus a smoothing preprocessing step it isn't reasonable to expect voxels individually carry all the information. The difference in voxels among groups should be, in fact, spread over neighboring voxels. So, cluster-wise comparison is performed, i.e. only differences between groups that form into clusters are considered. This cluster extent thresholding is the most popular multiple comparison correction technique in fMRI studies. The problem lies here. SPM and FSL depend on Gaussian random-field theory (RFT) for FWE-corrected voxelwise and clusterwise inference. However, RFT clusterwise inference depends on two additional assumptions. The first assumption is that the spatial smoothness of the fMRI signal is constant over the brain, and the second assumption is that the spatial autocorrelation function has a specific shape (a squared exponential) (30) In SPM at least you have to set a nominal FWE rate and also a cluster-defining threshold (CDT). Basically, SPM finds voxels highly correlated to the task and, after thresholding with the CDT, neighboring ones are aggregated into clusters. These clusters sizes are compared to the expected cluster extent from Random Field Theory (RFT) given the FWER set [1]. Random field theory requires the activity map to be smooth, to be a good lattice approximation to random fields. This is related to the amount of smoothing that is applied to the volumes. The smoothing also affects the assumption that the residuals are normally distributed, as smoothing, by the central limit theorem, will make the data more Gaussian. The authors have shown in [1] that the expected cluster sizes from RFT are really small when comparing with cluster extent thresholds obtained from random permutation testing (RPT). In their most recent paper, resting-state (another modality of fMRI, where participants are instructed to not think in anything in particular) data was used as if people performed a task during image acquisition, and the group comparison was performed voxel- and cluster-wise. The observed false positive error (i.e. when you observe differences in the signal response to a virtual task between groups) rate should be reasonably lower than the expected FWE rate set at$\alpha = 0.05\$. Redoing this analysis millions of times on randomly sampled groups with different paradigms showed most observed FWE rates to be higher than acceptable though.

@amoeba raised these two highly pertinent questions in the comments:

(1) The Eklund et al. PNAS paper talks about "nominal 5% level" of all the tests (see e.g. horizontal black line on Fig 1). However, CDT in the same figure is varying and can be e.g. 0.01 and 0.001. How does CDT threshold relate to the nominal type I error rate? I am confused by that. (2) Have you seen Karl Friston's reply http://arxiv.org/abs/1606.08199 ? I read it, but I am not quite sure what they are saying: do I see correctly that they agree with Eklund et al. but say that this is a "well known" issue?

(1) Good question. I actually reviewed my references, let's see if I can make it clearer now. Cluster-wise inference is based on the extent of clusters that form after a primary threshold (the CDT, which is arbitrary) is applied. In the secondary analysis a threshold on the number of voxels per cluster is applied. This threshold is based on the expected distribution of null cluster extents, which can be estimated from theory (e.g. RFT), and sets a nominal FWER. A good reference is [2].

(2) Thanks for this reference, didn't see it before. Flandin & Friston argue Eklund et al. corroborated RFT inference because they basically showed that respecting its assumptions (regarding CDT and smoothing) the results are unbiased. Under this light, the new results show different practices in the literature tend to bias the inference as it breaks down the assumptions of RFT.

## On the multiple comparisons

It's also well known many studies in neuroscience don't correct for multiple comparisons, estimates ranging from 10% to 40% of the literature. But these are not accounted by that claim, everyone knows these papers have fragile validity and possibly huge false positive rates.

## On the FWER in excess of 70%

The authors also reported a procedure that produces FWER in excess of 70%. This "folk"-procedure consists in applying the CDT to keep only highly significant clusters and then applying another arbitrarily chosen cluster-extent threshold (in number of voxels). This, sometimes called "set-inference", has weak statistical bases, and possibly generates the least trustworthy results.

## Previous reports

The same authors had already reported on problems with the validity of SPM [1] on individual analyses. There are also other cited works in this area.

Curiously, several reports on group- and individual-level analysis based on simulated data concluded the RFT threshold were, in fact, conservative. With recent advances in processing power though RPT can be performed much more easily on real data, showing great discrepancies with RFT.

## UPDATE: October 18th, 2017

A commentary on "Cluster Failure" has surfaced last June [3]. There Mueller et al. argue the results presented in Eklund et al might be due to a specific imaging preprocessing technique used in their study. Basically, they resampled the functional images to a higher resolution before smoothing (while probably not done by every researcher, this is a routine procedure in most fMRI analysis software). They also note that Flandin & Friston didn't. I actually got to see Eklund talk at the same month in the Organization for Human Brain Mapping (OHBM) Annual Meeting in Vancouver, but I don't remember any comments on this issue, yet it seems crucial to the question.

• @Qroid Yes for the first part, the assumption doesn't hold (and that's probably the cause for the good performance of the non-parametric permutation test). The clusters are clusters of voxels, i.e. neighboring voxels showing the same effect. There's a p-value to define a cluster (the cluster-defining threshold). Jul 25 '16 at 17:59
• This answer mostly focuses on whether it's 40000 or some other number, but I think it would be more interesting for many people here if you could summarize the main debate (what are the clusters? what's the problem with spatial correlations for hypothesis testing? did nobody really think of this before? etc.) Jul 25 '16 at 20:57
• Thanks again. After briefly looking at Woo et al. 2014, I am now sure anymore why Eklund et al. made it into PNAS and consequently produced such a sh*tstorm in the popular press and around the blogs. Weren't Woo et al. saying more or less the same thing? Here it is, right in their "highlights" blob: "Another pitfall is increased false positives when a liberal primary threshold is used". Jul 26 '16 at 14:13
• I see. So my understanding is that scientifically nothing much really happened now: the issue with liberal CDTs has been known for years, discussed in multiple papers, and shown in various simulations by various researchers. (But nevertheless, some researchers continued using such dangerously liberal CDTs.) Eklund et al. 2016 got lucky to be published in a "high profile" journal and boo! - everybody is now talking about it as if it's a revelation. Jul 26 '16 at 14:34
• @amoeba The neuroscience community needs a statistical crackdown, kinda like what happened in applied psychology (perhaps not so drastic as banning p-values). Many papers claiming statistical significance have no statistical rigour, people use the tools and the parameters that make "results appear". Jul 26 '16 at 14:39