13
$\begingroup$

There are obviously many ways to abuse statistical methods. Do you know of any examples of poor statistical practice that were first published as explicit advice (e.g. "you should use this method to ..."), in reputable academic journals that then went on to be repeatedly cited?

An example might be the 10 events per predictor rule that is often invoked for logistic or Cox PH regression models (LINK).

To be clear, I don't mean highly cited papers that happened to use poor stats methods - these are trivially common, unfortunately.

$\endgroup$
  • 3
    $\begingroup$ Are you looking for original publication in statistics journals? There is no end to statistical bad practice that is propagated in non-statistics journals (and when a reviewer points out that something is wrong, the authors will usually argue to leave it in "to tie our paper into previous research"). It may be hard to figure out any original publication for things like discretizing continuous outcomes, though, since bad ideas crop up independently. $\endgroup$ – S. Kolassa - Reinstate Monica Aug 24 at 9:33
  • $\begingroup$ I mean stated as explicit advice, e.g. "do this...". I've edited the question to clarify. Thanks. $\endgroup$ – D L Dahly Aug 24 at 10:30
  • 2
    $\begingroup$ You don't often see explicit "do this" commands in stats journals. You do see it in some application areas, particularly when written by people who are criticizing some problematic practice(s) (where they do sometimes say 'don't do A, do B' - but may give fairly dubious advice themselves. Is that the sort of thing you're after? I don't read journals in other areas all that much but I have seen some papers like that in the past. (Even if I could remember exactly where, though, I can't say I know whether any of them were highly cited). ... ctd $\endgroup$ – Glen_b -Reinstate Monica Aug 24 at 11:03
  • 1
    $\begingroup$ ctd... While not a paper, I can point to some dubious advice in a textbook that seems to be popular with people learning to do statistics for research in its application area. $\endgroup$ – Glen_b -Reinstate Monica Aug 24 at 11:19
  • 2
    $\begingroup$ Please explain what you mean by "statistical falsehood." This is not a standard concept in statistics, which rather concerns recommending procedures that are more or less appropriate for a given task. Yes, some procedures are known to be poorer than others, but it's difficult to construe their use as a "falsehood." By "falsehood" would you mean some kind of misleading interpretation, or advice to use an inadmissible procedure, or advice based on a mathematical error, or... what? $\endgroup$ – whuber Aug 24 at 13:45
4
$\begingroup$

R. A. Fisher, "The Arrangement of Field Experiments". Journal of the Ministry of Agriculture of Great Britain. 33: 503–513. 1926.

According to various sources on the internet, this paper is the origin of using $\alpha = 0.05$ as the significance threshold in an arbitrary statistical test.

... it is convenient to draw the line at about the level at which we can say: "Either there is something in the treatment, or a coincidence has occurred such as does not occur more than once in twenty trials."

... If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.

$\endgroup$
3
$\begingroup$

In econometrics, you can certainly find some example of propagated methods by well-known (and highly skilled) econometricians published in decent journals. I am not aware of a theoretical paper but Lalonde (1986) is quite famous for pointing out that currently used methods do not well: He compares for the same dataset experimental methods with observational ones and finds large differences in the field of (causal) treatment evaluation. There is a large literature which did propagate these non-experimental methods which have been used back then and which are often still used today.

Subsequently, there was (and I think still is) a debate about whether propensity score matching is a possible solution (see for example here).

Furthermore, there is a lot of controversy about instrumental variable estimation. The conclusions of highly cited original papers have been disputed. This is probably closest example to your question. Bound and Jaeger (1996, and subsequent papers) have questioned the findings of the well-known paper from Angrist and Krueger (1991; 2700 citations according to Google Scholar) which basically established the instrumental variable method in the applied econometrics literature.

There is also large debate about the appropriatness of so-called reduced form estimates to establish causality, see for example Imbens (2010).

Another big topic is of course about standard error. One can perhaps find a well-known paper propagating p-values. In econometrics, standard error for longer time-series have often been miscalculated (in the difference-in-difference design) due to wrong existing methods, see here. I am however not aware of an original highly-cited paper proposing these methods in that context but I am sure that you will find some examples in this area.

Sources:

Angrist, Joshua D., and Alan B. Keueger. "Does compulsory school attendance affect schooling and earnings?." The Quarterly Journal of Economics 106, no. 4 (1991): 979-1014.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan. "How much should we trust differences-in-differences estimates?." The Quarterly journal of economics 119, no. 1 (2004): 249-275.

Bound, John, and David A. Jaeger. On the Validity of Season of Birth as an Instrument in Wage Equations: A Comment on Angrist & Krueger's" Does Compulsory School Attendance Affect Scho. No. w5835. National Bureau of Economic Research, 1996.

Dehejia, Rajeev. "Practical propensity score matching: a reply to Smith and Todd." Journal of econometrics 125, no. 1-2 (2005): 355-364.

Imbens, Guido W. "Better LATE than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009)." Journal of Economic literature 48, no. 2 (2010): 399-423.

LaLonde, Robert J. "Evaluating the econometric evaluations of training programs with experimental data." The American economic review (1986): 604-620.*

$\endgroup$
1
$\begingroup$

I give a try (though not so strong):

The very useful [Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2), 317-372.] // already 1900 Google scholar citations// provides advice regarding the appropriate level of clustering of standard errors:

"The consensus is to be conservative and avoid bias and to use bigger and more aggregate clusters when possible, up to and including the point at which there is concern about having too few clusters."

However, [Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. (2017). When should you adjust standard errors for clustering? (No. w24003). National Bureau of Economic Research.] shows that "there is in fact harm in clustering at too aggregate a level". Please see page 1 of the later: https://economics.mit.edu/files/13927

Maybe you could also be able to make a stonger case starting from the two misconceptions highlighted by Abadie et al (2017).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.