Are there highly cited papers on statistics that have actually spread poor statistical practices?

Question

There are obviously many ways to abuse statistical methods. Do you know of any examples of poor statistical practice that were first published as explicit advice (e.g. "you should use this method to ..."), in reputable academic journals that then went on to be repeatedly cited?

An example might be the 10 events per predictor rule that is often invoked for logistic or Cox PH regression models (LINK).

To be clear, I don't mean highly cited papers that happened to use poor stats methods - these are trivially common, unfortunately.

Answer 1

In econometrics, you can certainly find some example of propagated methods by well-known (and highly skilled) econometricians published in decent journals. I am not aware of a theoretical paper but Lalonde (1986) is quite famous for pointing out that currently used methods do not well: He compares for the same dataset experimental methods with observational ones and finds large differences in the field of (causal) treatment evaluation. There is a large literature which did propagate these non-experimental methods which have been used back then and which are often still used today.

Subsequently, there was (and I think still is) a debate about whether propensity score matching is a possible solution (see for example here).

Furthermore, there is a lot of controversy about instrumental variable estimation. The conclusions of highly cited original papers have been disputed. This is probably closest example to your question. Bound and Jaeger (1996, and subsequent papers) have questioned the findings of the well-known paper from Angrist and Krueger (1991; 2700 citations according to Google Scholar) which basically established the instrumental variable method in the applied econometrics literature.

There is also large debate about the appropriatness of so-called reduced form estimates to establish causality, see for example Imbens (2010).

Another big topic is of course about standard error. One can perhaps find a well-known paper propagating p-values. In econometrics, standard error for longer time-series have often been miscalculated (in the difference-in-difference design) due to wrong existing methods, see here. I am however not aware of an original highly-cited paper proposing these methods in that context but I am sure that you will find some examples in this area.

Sources:

Angrist, Joshua D., and Alan B. Keueger. "Does compulsory school attendance affect schooling and earnings?." The Quarterly Journal of Economics 106, no. 4 (1991): 979-1014.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan. "How much should we trust differences-in-differences estimates?." The Quarterly journal of economics 119, no. 1 (2004): 249-275.

Bound, John, and David A. Jaeger. On the Validity of Season of Birth as an Instrument in Wage Equations: A Comment on Angrist & Krueger's" Does Compulsory School Attendance Affect Scho. No. w5835. National Bureau of Economic Research, 1996.

Dehejia, Rajeev. "Practical propensity score matching: a reply to Smith and Todd." Journal of econometrics 125, no. 1-2 (2005): 355-364.

Imbens, Guido W. "Better LATE than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009)." Journal of Economic literature 48, no. 2 (2010): 399-423.

LaLonde, Robert J. "Evaluating the econometric evaluations of training programs with experimental data." The American economic review (1986): 604-620.*

Answer 2

R. A. Fisher, "The Arrangement of Field Experiments". Journal of the Ministry of Agriculture of Great Britain. 33: 503–513. 1926.

According to various sources on the internet, this paper is the origin of using $\alpha = 0.05$ as the significance threshold in an arbitrary statistical test.

... it is convenient to draw the line at about the level at which we can say: "Either there is something in the treatment, or a coincidence has occurred such as does not occur more than once in twenty trials."

... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.

Answer 3

I give a try (though not so strong):

The very useful [Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2), 317-372.] // already 1900 Google scholar citations// provides advice regarding the appropriate level of clustering of standard errors:

"The consensus is to be conservative and avoid bias and to use bigger and more aggregate clusters when possible, up to and including the point at which there is concern about having too few clusters."

However, [Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. (2017). When should you adjust standard errors for clustering? (No. w24003). National Bureau of Economic Research.] shows that "there is in fact harm in clustering at too aggregate a level". Please see page 1 of the later: https://economics.mit.edu/files/13927

Maybe you could also be able to make a stonger case starting from the two misconceptions highlighted by Abadie et al (2017).