What are some examples of anachronistic practices in statistics?

Question

I am referring to practices that still maintain their presence, even though the problems (usually computational) they were designed to cope with have been mostly solved.

For example, Yates' continuity correction was invented to approximate Fisher's exact test with $\chi^2$ test, but it is no longer practical since software can now handle Fisher's test even with large samples (I know this may not be a good example of "maintaining its presence", since textbooks, like Agresti's Categorical Data Analysis, often acknowledge that Yates' correction "is no longer needed").

What are some other examples of such practices?

Answer 1

It's strongly arguable that the use of threshold significance levels such as $P = 0.05$ or $P = 0.01$ is a historical hangover from a period when most researchers depended on previously calculated tables of critical values. Now good software will give $P$-values directly. Indeed, good software lets you customise your analysis and not depend on textbook tests.

This is contentious if only because some significance testing problems do require decisions, as in quality control where accepting or rejecting a batch is the decision needed, followed by an action either way. But even there the thresholds to be used should grow out of a risk analysis, not depend on tradition. And often in the sciences, analysis of quantitative indications is more appropriate than decisions: thinking quantitatively implies attention to sizes of $P$-values and not just to a crude dichotomy, significant versus not significant.

I will flag that I here touch on an intricate and controversial issue which is the focus of entire books and probably thousands of papers, but it seems a fair example for this thread.

Answer 2

One method that I think many visitors of this site will agree with me on is stepwise regression. It's still done all the time, but you don't have to search far for experts on this site saying deploring its use. A method like LASSO is much preferred.

Answer 3

My view is that at least in (applied) econometrics, it is more and more the norm to use the robust or empirical covariance matrix rather than the "anachronistic practice" of relying (asymptotically) on the correct specification of the covariance matrix. This is of course not without controversy: see some of the answers I linked here at CrossValidated, but it is certainly a clear trend.

Examples include heteroscedasticity-robust standard error (Eicker-Huber-White standard errors). Some researchers such as Angrist and Pischke apparently advise always using heteroscedasticity-robust standard error rather than the "anachronistic" procedure to use normal standard error as default and check whether the assumption $E[uu'] = \sigma^2 I_n$ is warranted.

Other examples include panel data, Imbens and Wooldridge write for example in their lecture slides argue against using the random effects variance covariance matrix (implicitly assuming some misspecification in the variance component as default):

Fully robust inference is available and should generally be used. (Note: The usual RE variance matrix, which depends only on $\sigma_c^2$and $\sigma_u^2$, need not be correctly specified! It still makes sense to use it in estimation but make inference robust.)

Using generalized linear models (for distributions which belong to the exponential family), often it is advised to use always the so-called sandwich estimator rather than relying on correct distributional assumptions (the anachronistic practice here): see for example this answer or Cameron referring to count data because pseudo-maximum likelihood estimation can be quite flexible in the case of misspecification (e.g. using Poisson if negative binomial would be correct).

Such [White] standard error corrections must be made for Poisson regression, as they can make a much bigger difference than similar heteroskedasticity corrections for OLS.

Greene writes in his textbook in Chapter 14 (available on his website) for example with a critical note and goes more into detail about the advantages and disadvantages of this practice:

There is a trend in the current literature to compute this [sandwich] estimator routinely, regardless of the likelihood function.* [...] *We do emphasize once again that the sandwich estimator, in and of itself, is not necessarily of any virtue if the likelihood function is misspecified and the other conditions for the M estimator are not met.

Answer 4

A method that is unnecessarily used all the time is the Bonferroni correction to p-values. While multiple comparisons is as big an issue as it ever was, the Bonferroni correction is essentially obsolete for p-values: for any situation in which the Bonferroni correction is valid, so is the Holm-Bonferroni, which will have strictly higher power under the alternative if $m > 1$, where $m$ is the number of hypothesis tested (equality at $m = 1$).

I think the reason for the persistence of the Bonferroni correction is the ease of mental use (i.e. p = 0.004 with $m = 30$ is easily adjusted to 0.12, while Holm-Bonferroni requires sorting of p-values).

Answer 5

Most anachronistic practices are probably due to the way statistics is taught and the fact that analyses are run by huge numbers of people who have only taken a couple of basic classes. We often teach a set of standard statistical ideas and procedures because they form a logical sequence of increasing conceptual sophistication that makes sense pedagogically (cf., How can we ever know the population variance?). I'm guilty of this myself: I occasionally teach stats 101 and 102, and I constantly say, 'there's a better way to do this, but it's beyond the scope of this class'. For those students who don't go on beyond the introductory sequence (almost all), they are left with basic, but superseded, strategies.

1. For a stats 101 example, probably the most common anachronistic practice is to test some assumption and then run a traditional statistical analysis because the test was not significant. A more modern / advanced / defensible approach would be to use a method robust to that assumption from the start. Some references for more information:

   • How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
   • Is normality testing 'essentially useless'?
     

2. For stats 102 examples, any number of modeling practices have been outmoded:

   • Transforming $Y$ to achieve normality of residuals for getting reliable $p$-values vs. bootstrapping.
   • Transforming $Y$ to achieve homoscedasticity instead of using a sandwich estimator, etc.
   • Using a higher-order polynomial to capture curvature vs. cubic splines.
   • Assessing models intended for prediction using $p$-values and in-sample goodness of fit metrics like $R^2$ instead of cross-validation.
   • With repeated measures data, categorizing a continuous variable so that rmANOVA can be used or averaging multiple measurements vs. using a linear mixed model.
   • Etc.

The point in all these cases is that people are doing what was taught first in an introductory class because they simply don't know more advanced and appropriate methods.

Answer 6

Paying licensing fees for high-quality statistical software systems. #R

Answer 7

A very interesting example are unit root tests in econometrics. While there are plenty of choices available to test against or for a unit root in the lag polynomial of a time series (e.g., the (Augmented) Dickey Fuller Test or the KPSS test), the problem can be circumvented completely when one uses Bayesian analysis. Sims pointed this out in his provocative paper titled Understanding Unit Rooters: A Helicopter Tour from 1991.

Unit root tests remain valid and used in econometrics. While I personally would attribute this mostly to people being reluctant to adjust to Bayesian practices, many conservative econometricians defend the practice of unit root tests by saying that a Bayesian view of the world contradicts the premise of econometric research. (That is, economists think of the world as a place with fixed parameters, not random parameters that are governed by some hyperparameter.)

Answer 8

Teaching/conducting two-tailed tests for difference without simultaneously testing for equivalence in the frequentist realm of hypothesis testing is a deep commitment to confirmation bias.

There's some nuance, in that an appropriate power analysis with thoughtful definition of effect size can guard against this and provide more or less the same kinds of inferences, but (a) power analyses are so often ignored in presenting findings, and (b) I have never seen a power analysis for, for example, each coefficient estimated for each variable in a multiple regression, but it is straightforward to do so for combined tests for difference and tests for equivalence (i.e. relevance tests).

Answer 9

Using a Negative Binomial model rather than a (robust) Poisson model to identify a parameter of interest in a count variable, only because there is over-dispersion ?

See as a reference: https://blog.stata.com/2011/08/22/use-poisson-rather-than-regress-tell-a-friend/

The proof that Poisson is more robust in the case of fixed-effects is quite recent as it is offen made reference to: Wooldridge, J. M., “Distribution-Free Estimation of Some Nonlinear Panel Data Models,” Journal of Econometrics 90 (1999), 77–97.

Answer 10

Here are a few anachronisms:

• The neoplatonic assumption that there is a single, "true" population out there in the theoretical ether that is eternal, fixed and unmoving against which our imperfect samples can be evaluated does little to advance learning and knowledge.

• The reductionism inherent in mandates such as Occam's Razor is inconsistent with the times. OR can be summarized as, "Among competing hypotheses, the one with the fewest assumptions should be selected." Alternatives include Epicurus' Principle of Multiple Explanations, which roughly states, "If more than one theory is consistent with the data, keep them all."

• The whole peer-review system is desperately in need of an overhaul.

* Edit *

• With massive data containing tens of millions of features, there is no longer need for a variable selection phase.

• In addition, inferential statistics are meaningless.