# Statistics published in academic papers

I read a lot of evolutionary/ecological academic papers, sometimes with the specific aim of seeing how statistics are being used 'in the real world' outside of the textbook. I normally take the statistics in papers as gospel and use the papers to help in my statistical learning. After all, if a paper has taken years to write and has gone through rigorous peer review, then surely the statistics are going to be rock solid? But in the past few days, I've questioned my assumption, and wondered how often the statistical analysis published in academic papers is suspect? In particular, it might be expected that those in fields such as ecology and evolution have spent less time learning statistics and more time learning their fields.

How often do people find suspect statistics in academic papers?

-
–  Scortchi Apr 2 '14 at 8:37
Reviewers are often people that don't know much more about statistics than those writing the paper, so it can often be easy to publish poor statistics. –  Behacad Apr 2 '14 at 10:17
Getting a paper published is the first step towards its acceptance by the scientific community, not the last. Most published papers will have significant flaws in some area, the use of statistics is no exception. –  Dikran Marsupial Apr 3 '14 at 13:12
Your assumption that papers "take years to write" is way off the mark. Collecting data might take a long time but analyzing the data and writing up is typically weeks rather than years. –  David Richerby Apr 3 '14 at 14:07
It is nowadays well known that statistics in many psychology and medicine papers is questionable at the least, plain wrong or not even that quite often. The poor-man usage of p-values and NHST is a prominent example of the problems, see this note. –  Quartz Apr 9 '14 at 11:26

After all, if a paper has taken years to write and has gone through rigorous peer review, then surely the statistics are going to be rock solid?

My experience of reading papers that attempt to apply statistics across a wide variety of areas (political science, economics, psychology, medicine, biology, finance, actuarial science, accounting, optics, astronomy, and many, many others) is that the quality of the statistical analysis may be anywhere on the spectrum from excellent and well done to egregious nonsense. I have seen good analysis in every one of the areas I have mentioned, and pretty poorly done analysis in almost all of them.

Some journals are generally pretty good, and some can be more like playing darts with a blindfold - you might get most of them not too terribly far off the target, but there's going to be a few in the wall, the floor and the ceiling. And maybe the cat.

I don't plan on naming any culprits, but I will say I have seen academic careers built on faulty use of statistics (i.e. where the same mistakes and misunderstandings were repeated in paper after paper, over more than a decade).

So my advice is let the reader beware; don't trust that the editors and peer reviewers know what they're doing. Over time you may get a good sense of which authors can generally be relied on to not do anything too shocking, and which ones should be treated especially warily. You may get a sense that some journals typically have very high standard for their stats.

But even a typically good author can make a mistake, or referees and editors can fail to pick up errors they might normally find; a typically good journal can publish a howler.

[Sometimes, you'll even see really bad papers win prizes or awards... which doesn't say much for the quality of the people judging the prize, either.]

I wouldn't like to guess what the fraction of "bad" stats I might have seen (in various guises, and at every stage from defining the question, design of the study, data collection, data management, ... right through to analysis and conclusions), but it's not nearly small enough for me to feel comfortable.

I could point to examples, but I don't think this is the right forum to do that. (It would be nice if there was a good forum for that, actually, but then again, it would likely become highly "politicized" quite quickly, and soon fail to serve its purpose.)

I've spent some time trawling through PLOS ONE ... and again, not going to point at specific papers. Some things I noticed: it looks like a large proportion of papers have stats in them, probably more than half having hypothesis tests. The main dangers seem to be lots of tests, either with high $\alpha$ like 0.05 on each (which is not automatically a problem as long as we understand that quite a few really tiny effects might show up as significant by chance), or an incredibly low individual significance level, which will tend to give low power. I also saw a number of cases where about half a dozen different tests were apparently applied to resolving exactly the same question. This strikes me as a generally bad idea. Overall the standard was pretty good across a few dozen papers, but in the past I have seen an absolutely terrible paper there.

[Perhaps I could indulge in just one example, indirectly. This question asks about one doing something quite dubious. It's far from the worst thing I've seen.]

On the other hand, I also see (even more frequently) cases where people are forced to jump through all kinds of unnecessary hoops to get their analysis accepted; perfectly reasonable things to do are not accepted because there's a "right" way to do things according to a reviewer or an editor or a supervisor, or just in the unspoken culture of a particular area.

-
"Caveat lector", given the increasing number of open-access journals? –  Scortchi Apr 2 '14 at 9:41
@scortchi I decided to avoid the issue altogether by simply writing in English. It's an improvement. –  Glen_b Apr 2 '14 at 10:08
Without naming specific culprits, I think faculty.vassar.edu/abbaird/about/publications/pdfs/… deserves a mention. To prove a point about misuse of statistics in their field, they used a widely used statistical protocol to analyse the results of an fMRI scan of a dead salmon. They found "statistically significant" brain activity. statisticsdonewrong.com also makes interesting reading. –  James_pic Apr 2 '14 at 10:51
@James_pic, had to join to +1 that comment for the statisticsdonewrong link; the discussion of the base rate fallacy is particularly interesting. –  Dan Bryant Apr 2 '14 at 13:34
@KennyPeanuts: Neither - just pointing out that nowadays many lectores aren't even indirectly emptores. –  Scortchi Apr 2 '14 at 14:19

I respect @Glen_b's stance on the right way to answer here (and certainly don't intend to detract from it), but I can't quite resist pointing to a particularly entertaining example that's close to my home. At the risk of politicizing things and doing this question's purpose a disservice, I recommend Wagenmakers, Wetzels, Boorsboom, and Van Der Maas (2011). I cited this in a related post on the Cognitive Sciences beta SE (How does cognitive science explain distant intentionality and brain function in recipients?), which considers another example of "a dart hitting the cat". Wagenmakers and colleagues' article comments directly on a real "howler" though: it was published in JPSP (one of the biggest journals in psychology) a few years ago. They also argue more generally in favor of Bayesian analysis and that:

In order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal.

I probably don't need to tell you that this didn't exactly come across as preaching to the choir. FWIW, there is a rebuttal as well (as there always seems to be between Bayesians and frequentists; (Bem, Utts, & Johnson, 2011), but I get the feeling that it didn't exactly checkmate the debate.

Psychology as a scientific community has been on a bit of a replication kick recently, partly due to this and other high-profile methodological shortcomings. Other comments here point to cases similar to what were once known as voodoo correlations in social neuroscience (how's that for politically incorrect BTW? the paper has been retitled; Vul, Harris, Winkielman, & Pashler, 2009). That too attracted its rebuttal, which you can check out for more debate of highly debatable practices.

For even more edutainment at the (more depersonalized) expense of (pseudo)statisticians behaving badly, see our currently 8th-most-upvoted question here on CV with another (admittedly) politically incorrect title, "What are common statistical sins?" Its OP @MikeLawrence attributes his inspiration to his parallel study of psychology and statistics. It's one of my personal favorites, and its answers are very useful for avoiding the innumerable pitfalls out there yourself.

On the personal side, I've been spending much of my last five months here largely because it's amazingly difficult to get rock-solid statistics on certain data-analytic questions. Frankly, peer review is often not very rigorous at all, especially in terms of statistical scrutiny of research in younger sciences with complex questions and plenty of epistemic complications. Hence I've felt the need to take personal responsibility for polishing the methods in my own work.

While presenting my dissertation research, I got a sense of how important personal responsibility for statistical scrutiny is. Two exceptional psychologists at my alma mater interjected that I was committing one of the most basic sins in my interpretations of correlations. I'd thought myself above it, and had lectured undergrads about it several times already, but I still went there, and got called out on it (early on, thank heavens). I went there because research I was reviewing and replicating went there! Thus I ended up adding several sections to my dissertation that called out those other researchers for assuming causality from quasi-experimental longitudinal studies (sometimes even from cross-sectional correlations) and ignoring alternative explanations prematurely.

My dissertation was accepted without revisions by my committee, which included another exceptional psychometrician and the soon-to-be-president of SPSP (which publishes JPSP), but to be frank once more, I'm not bragging in saying this. I've since managed to poke several rabbit holes in my own methods despite passing the external review process with perfectly good reviewers. I've now fallen into the deep end of stats in trying to plug them with methods more appropriate for predictive modeling of Likert ratings like SEM, IRT, and nonparametric analysis (see Regression testing after dimension reduction). I'm opting voluntarily to spend years on a paper that I could probably just publish as-is instead...I think I even have a simulation study left to do before I can proceed conscientiously.

Yet I emphasize that this is optional – maybe even overzealous and a costly luxury amidst the publish-or-perish culture that often emphasizes quantity over quality in early-career work records. Misapplication of parametric models for continuous data to assumption-violating distributions of ordinal data is all too common in my field, as is the misinterpretation and misrepresentation of statistical significance (see Accommodating entrenched views of p-values). I could totally get away with it (in the short term)...and it's not even all that hard to do better than that. I suppose I have several recent years of amazing advances in R programs to thank for that though! Here's hoping the times are changing.

References
· Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101(4), 716–719. Retrieved from http://deanradin.com/evidence/Bem2011.pdf.
· Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4(3), 274–290. Retrieved from http://www.edvul.com/pdf/VulHarrisWinkielmanPashler-PPS-2009.pdf.
· Wagenmakers, E. J., Wetzels, R., Borsboom, D., & Van der Maas, H. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426–432. Retrieved from http://mpdc.mae.cornell.edu/Courses/MAE714/Papers/Bem6.pdf.

-
If you enjoyed "Feeling the Future", then you might like Witztum et al. (1994), "Equidistant Letter Sequences in the Book of Genesis", Statist. Sci., 9,3. It attracted the inevitable scoffers & nay-sayers: McKay et. al. (1999), "Solving the Bible Code Puzzle", Statist. Sci., 14,2. –  Scortchi Apr 2 '14 at 15:56
@Scortchi: thanks for the reference, and amoeba: thanks for the context. I don't see the claim in Witzum et al. that McKay et al. scoff at in their abstract, but they sure point out a lot of other serious flaws. Good stuff. "Whereas real data may confound the expectations of scientists even when their hypotheses are correct, those whose experiments are systematically biased towards their expectations are less often disappointed (Rosenthal, 1976)." That's one of the guys who called me out on causal inference based on quasi-experiments...a truly great psychologist. Bem has some cred too though. –  Nick Stauner Apr 2 '14 at 23:14
+1 Excellent post. "how important personal responsibility for statistical scrutiny is" -- I must applaud. Ultimately, this is where responsibility must lie, as onerous as that may be for someone already trying to get work done in an area of research to which they wish to apply statistics. –  Glen_b Apr 3 '14 at 4:09
@NickStauner: McKay et al. say in their abstract that Witzum et al. claim "the Hebrew text of the Book of Genesis encodes events which did not occur until millennia after the text was written". Slight hyperbole perhaps, as it's just over two millenia at most between the writing of the Torah & the birth-date of the last rabbi from their list, but a fair enough summary. (I suppose you could also see the Witztum et al. paper as evidence for recent authorship of the Book of Genesis, though as far as I know no-one has done.) –  Scortchi Apr 3 '14 at 11:31
Yeah, I guess I couldn't understand Witzum et al. well enough to recognize that they were making that claim. For once I suppose I could be thankful for the authors' obtuse writing...It comes across as a little more interesting at face value because the most prominent claim is that the pattern is not due to chance, not what the pattern is supposedly due to in their opinion. It could've invited more interesting interpretations like yours had it not overreached as McKay et al. say it does...at least until McKay et al. shot them down on methodological grounds, leaving nothing worth interpreting. –  Nick Stauner Apr 3 '14 at 11:38

I recall at University being ask by a few final year social science students on different occasions (one of them got a 1st) how to work out an average for their project that had had a handful of data points. (So they were not having problem with using software, just with the concept of how to do the maths with a calculator.)

They just give me blank looks when I ask them what type of average they wanted.

Yet they all felt a need to put some stats in their report, as it was the done thing – I expect they have all read 101 papers that had stats without thinking about what the stats meant if anything.

It is clear that the researcher that taught them over the 3 years did not care about the correctness of stats enough to distil any understanding into the students.

(I was a computer Sci student at the time. I am posting this as an answer as it is a bit long for a comment.)

-
Students are a whole other barrel of monkeys, IMO. I wouldn't blame the teacher immediately for their lack of understanding without further evidence...but if it's as clear as you say that the teacher is to blame, I wouldn't be surprised either. –  Nick Stauner Apr 3 '14 at 20:01
@NickStauner, I blame the teacher for not caring enough about stats; if they cared there would be at least one question on each exam paper that needed some understanding of stats, at the level of “How to Lie with Statistics”. I don’t care if social science students know how to do the calc, but they should know how not to be mislead. –  Ian Ringrose Apr 3 '14 at 21:35
Agreed that they should know, but there's no guaranteeing they'll get that question right! –  Nick Stauner Apr 3 '14 at 21:38
@NickStauner, Yes, but you only get what the measure, so you will not get students that understand anything about stats unless you put it in the exams. –  Ian Ringrose Apr 3 '14 at 22:04
Again, I tend to give teachers less credit for student outcomes. Plenty of students (okay, maybe not "plenty", but some) will care enough to learn for its own sake, and some will come to class already knowing much of the material. Forgive me if I interpret your comment too absolutely though; I would agree that it is often a necessary evil to force motivation to learn onto students, and that testing is a better way to learn than rote, repetitive studying/lecturing. –  Nick Stauner Apr 3 '14 at 22:54

Any paper that disproves the nil null hypothesis is using worthless statistics (the vast majority of what I have seen). This process can provide no information not already provided by the effect size. Further it tells us nothing about whether a significant result is actually due to the cause theorized by the researcher. This requires thoughtful investigation of the data for evidence of confounds. Most often, if present, the strongest of this evidence is even thrown away as "outliers".

I am not so familiar with evolution/ecology, but in the case of psych and medical research I would call the level of statistical understanding "severely confused" and "an obstacle to scientific progress". People are supposed to be disproving something predicted by their theory, not the opposite of it (zero difference/effect).

There have been thousands of papers written on this topic. Look up NHST hybrid controversy.

Edit: And I do mean the nill null hypothesis significance test has a maximum of zero scientific value. This person hits the nail on the head:

http://www.johnmyleswhite.com/notebook/2012/05/18/criticism-4-of-nhst-no-mechanism-for-producing-substantive-cumulative-knowledge/

Also: Paul Meehl. 1967. Theory Testing in Psychology and Physics: A Methodological Paradox

Edit 3:

If someone has arguments in favor of the usefulness of strawman NHST that do not require thinking "reject the hypothesis that the rate of warming is the same, but DO NOT take this to imply that the rate of warming is the not same" is a rational statement, I would welcome your comments.

Edit 4:

What did Fisher mean by the following quote? Does it suggest that he thought "If model/theory A is incompatible with the data, we can say A is false, but nothing about whether not A is true"?

"it is certain that the interest of statistical tests for scientific workers depends entirely from their use in rejecting hypotheses which are thereby judged to be incompatible with the observations."

...

It would, therefore, add greatly to the clarity with which the tests of significance are regarded if it were generally understood that tests of significance, when used accurately, are capable of rejecting or invalidating hypotheses, in so far as these are contradicted by the data; but that they are never capable of establishing them as certainly true

Karl Pearson and R. A. Fisher on Statistical Tests: A 1935 Exchange from Nature

Is it that he assumed people would only try to invalidate plausible hypotheses rather than strawmen? Or am I wrong?

-
"This process can provide no information not already provided by the effect size." this is incorrect, the p-value provides some information about how unusual this effect size would be under the null hypothesis, thus it provides an element of calibration of effect size. Don't misunderstand me, I think Bayes factors are more useful, but it is hyperbole to say that the p-value is a worthless statistic. –  Dikran Marsupial Apr 3 '14 at 13:09
"I find that all patterns I (and others) notice are worth mentioning" this is exactly the problem that arises in the discussion of climate on blogs, the human eye is very good at seeing patterns in data that turn out to be just noise, and it does the signal-to-noise ratio in the debate no good at all not to have some hurdle for an idea to get over before posting it on a blog! It is one area of science where the statistics are often very poor. –  Dikran Marsupial Apr 3 '14 at 15:27
Livid, I gave you a concrete example of where performing an appropriate NHST with a "straw man" H0 would be beneficial to the discussion of a scientific topic. That provides a clear counterexample that demonstrates your view to be incorrect - NHSTs, as flawed as they are, do nevertheless perform a useful function in science and statistics. Now if you can demonstrate that my counterexample is correct, that may go some way towards resolving the issue. –  Dikran Marsupial Apr 4 '14 at 8:18
@Livid, NHST performs a scientifically and statistically, not socially desirable function (though not optimally) and it doesn't set an arbitrary obstacle, the hurdle is generally defined by its opposition to H1 and it doesn't involve committing "affirming the consequent fallacies" as rejecting H0 does not imply that H1 is true. So no, it isn't accurate. –  Dikran Marsupial Apr 4 '14 at 17:59
You are missing the point. If you have a low hurdle, then nobody is surprise if you can negotiate it successfully. However if you have a low hurdle, but you still can't get over it, that tells you something. As I have repeatedly said, rejecting the null does not imply that H1 is true, so rejecting H0 doesn't mean that there definitely is a pause, it doesn't tell you why there has been a pause. But if you can't get over the hurdle of being able to reject H0, it suggests that perhaps there is insufficient evidence to assert H1 as fact (which is what is happening in this instance). –  Dikran Marsupial Apr 4 '14 at 18:51