# What are some examples of anachronistic practices in statistics?

I am referring to practices that still maintain their presence, even though the problems (usually computational) they were designed to cope with have been mostly solved.

For example, Yates' continuity correction was invented to approximate Fisher's exact test with $\chi^2$ test, but it is no longer practical since software can now handle Fisher's test even with large samples (I know this may not be a good example of "maintaining its presence", since textbooks, like Agresti's Categorical Data Analysis, often acknowledge that Yates' correction "is no longer needed").

What are some other examples of such practices?

• I'm actually not so sure that the chi-squared test was made obsolete by the availability of computing power to perform Fisher's exact test, e.g. are your marginals truly fixed? See this answer to another question by @gung, for instance. (I'm pretty sure that we have a thread discussing the issue in more detail but I can't find it as we have a lot of "should I use chi-squared or should I use Fisher's exact test" question that show up when I search!) – Silverfish Jun 19 '16 at 1:48
• @Silverfish: I didn't mean $\chi^2$ was made obsolete, only Yates' correction was. I believe studies have shown that Yates' correction are too conservative when the marginals are not fixed. Michael Haber's article The Continuity Correction and Statistical Testing provided a review. – Francis Jun 19 '16 at 2:15
• @Silverfish, this is probably what you're after: Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test? – gung - Reinstate Monica Jun 19 '16 at 2:15
• using OLS instead of LAD? – PatrickT Jun 19 '16 at 18:31
• @PatrickT: I have a lot of trouble calling OLS anachronistic. Sure, there's special cases when LAD is clearly superior...but same can be said in the other direction. – Cliff AB Jun 19 '16 at 20:10

It's strongly arguable that the use of threshold significance levels such as $P = 0.05$ or $P = 0.01$ is a historical hangover from a period when most researchers depended on previously calculated tables of critical values. Now good software will give $P$-values directly. Indeed, good software lets you customise your analysis and not depend on textbook tests.

This is contentious if only because some significance testing problems do require decisions, as in quality control where accepting or rejecting a batch is the decision needed, followed by an action either way. But even there the thresholds to be used should grow out of a risk analysis, not depend on tradition. And often in the sciences, analysis of quantitative indications is more appropriate than decisions: thinking quantitatively implies attention to sizes of $P$-values and not just to a crude dichotomy, significant versus not significant.

I will flag that I here touch on an intricate and controversial issue which is the focus of entire books and probably thousands of papers, but it seems a fair example for this thread.

• Great example! For reference, this thread is worth mentioning: Regarding p-values, why 1% and 5%? Why not 6% or 10%? – Francis Jun 18 '16 at 11:39
• @ J. M. I'm 95% confident you are correct, although I am not 99% confident. – Mark L. Stone Jun 18 '16 at 17:43
• Actually, I'm not sure this is a great example. While it's true that it's much easier to test things at $\alpha = 0.038561$ than it used to, I've never seen a good argument for why you would want to, outside special cases (i.e. quality control), in which case I don't know that arbitrary significance levels are still used. – Cliff AB Jun 18 '16 at 20:00
• @CliffAB I don't think the main point of an exact P-value is that you then decide that it constitutes the critical level you want to adopt for a decision. I certainly am not suggesting or advocating that. Part of the argument here is not just that 0.05 and 0.01 are at best conventional levels but that tests provide one way to assess strength of evidence against a null hypothesis rather than to make a binary decision. In practice 0.05 and 0.01 levels remain very heavily used in many fields. – Nick Cox Jun 18 '16 at 22:05
• @Nick Cox And don't forget the 0.1 level for the laid-back, mellow crowd. – Mark L. Stone Jun 18 '16 at 22:07

One method that I think many visitors of this site will agree with me on is stepwise regression. It's still done all the time, but you don't have to search far for experts on this site saying deploring its use. A method like LASSO is much preferred.

• HA!! You're recommending the replacement of an anachronism (stepwise regression) with the next generation anachronism (LASSO), which is an anachronism in its own time, whose adherents do not yet realize it. See stats.stackexchange.com/questions/162861/… . – Mark L. Stone Jun 18 '16 at 20:19
• @MarkL.Stone: Hey man, at least it's 20 years in the right direction. I'm not so familiar with these methods, so I would have to read up on them before I could give them my endorsement. – Cliff AB Jun 18 '16 at 20:31
• After quickly reading the article, I'm a little hesitant to decide that LASSO is officially outdated, although it's clearly not always the optimal choice. Maybe in 5 years I'll be more comfortable calling LASSO obsolete. – Cliff AB Jun 18 '16 at 21:03
• @amoeba: I think Mark is referring to the practice of using LASSO as a tool for best subset regression. For example, I vaguely recall reading someone discuss first fitting LASSO, and then refitting a non-penalized model using the non-zero regression parameters. Best subset regression may be a more direct way of doing this (though as you say, it's not clear that this is a good idea even if it's what the analyst wants to do). – Cliff AB Jun 22 '16 at 23:47
• ...and the paper presents at least one situation (i.e. simulation under certain parameters) where it clearly out performs LASSO, although I think we all know exactly how seriously we should take such results on by themselves. – Cliff AB Jun 22 '16 at 23:49

My view is that at least in (applied) econometrics, it is more and more the norm to use the robust or empirical covariance matrix rather than the "anachronistic practice" of relying (asymptotically) on the correct specification of the covariance matrix. This is of course not without controversy: see some of the answers I linked here at CrossValidated, but it is certainly a clear trend.

Examples include heteroscedasticity-robust standard error (Eicker-Huber-White standard errors). Some researchers such as Angrist and Pischke apparently advise always using heteroscedasticity-robust standard error rather than the "anachronistic" procedure to use normal standard error as default and check whether the assumption $E[uu'] = \sigma^2 I_n$ is warranted.

Other examples include panel data, Imbens and Wooldridge write for example in their lecture slides argue against using the random effects variance covariance matrix (implicitly assuming some misspecification in the variance component as default):

Fully robust inference is available and should generally be used. (Note: The usual RE variance matrix, which depends only on $\sigma_c^2$and $\sigma_u^2$, need not be correctly specified! It still makes sense to use it in estimation but make inference robust.)

Using generalized linear models (for distributions which belong to the exponential family), often it is advised to use always the so-called sandwich estimator rather than relying on correct distributional assumptions (the anachronistic practice here): see for example this answer or Cameron referring to count data because pseudo-maximum likelihood estimation can be quite flexible in the case of misspecification (e.g. using Poisson if negative binomial would be correct).

Such [White] standard error corrections must be made for Poisson regression, as they can make a much bigger difference than similar heteroskedasticity corrections for OLS.

Greene writes in his textbook in Chapter 14 (available on his website) for example with a critical note and goes more into detail about the advantages and disadvantages of this practice:

There is a trend in the current literature to compute this [sandwich] estimator routinely, regardless of the likelihood function.* [...] *We do emphasize once again that the sandwich estimator, in and of itself, is not necessarily of any virtue if the likelihood function is misspecified and the other conditions for the M estimator are not met.

• Interesting, but the question is what is anachronistic, not what is now increasingly standard, so the answer has to be inverted. – Nick Cox Jun 18 '16 at 11:08
• Hi Nick, thanks for your comment (and your edits), I modified the text to highlight what is the anachronistic practice, I hope it becomes a little bit clearer. I did not invert the whole text, since the former practice is close to doing nothing specially about the standard error. – Arne Jonas Warnke Jun 18 '16 at 12:12
• In some cases it is not natural and not possible to use robust alternatives, say time series. So I think it is not becoming "more popular" but just "more popular in some areas". – Henry.L Dec 14 '16 at 2:43

A method that is unnecessarily used all the time is the Bonferroni correction to p-values. While multiple comparisons is as big an issue as it ever was, the Bonferroni correction is essentially obsolete for p-values: for any situation in which the Bonferroni correction is valid, so is the Holm-Bonferroni, which will have strictly higher power under the alternative if $m > 1$, where $m$ is the number of hypothesis tested (equality at $m = 1$).

I think the reason for the persistence of the Bonferroni correction is the ease of mental use (i.e. p = 0.004 with $m = 30$ is easily adjusted to 0.12, while Holm-Bonferroni requires sorting of p-values).

• Came here to post this. Also: am not convinced there is any situation where FWER would be preferred to the even newer FDR methods (because of scalability and adaptability). – Alexis May 24 '18 at 15:09

Most anachronistic practices are probably due to the way statistics is taught and the fact that analyses are run by huge numbers of people who have only taken a couple of basic classes. We often teach a set of standard statistical ideas and procedures because they form a logical sequence of increasing conceptual sophistication that makes sense pedagogically (cf., How can we ever know the population variance?). I'm guilty of this myself: I occasionally teach stats 101 and 102, and I constantly say, 'there's a better way to do this, but it's beyond the scope of this class'. For those students who don't go on beyond the introductory sequence (almost all), they are left with basic, but superseded, strategies.

1. For a stats 101 example, probably the most common anachronistic practice is to test some assumption and then run a traditional statistical analysis because the test was not significant. A more modern / advanced / defensible approach would be to use a method robust to that assumption from the start. Some references for more information:

2. For stats 102 examples, any number of modeling practices have been outmoded:

• Transforming $Y$ to achieve normality of residuals for getting reliable $p$-values vs. bootstrapping.
• Transforming $Y$ to achieve homoscedasticity instead of using a sandwich estimator, etc.
• Using a higher-order polynomial to capture curvature vs. cubic splines.
• Assessing models intended for prediction using $p$-values and in-sample goodness of fit metrics like $R^2$ instead of cross-validation.
• With repeated measures data, categorizing a continuous variable so that rmANOVA can be used or averaging multiple measurements vs. using a linear mixed model.
• Etc.

The point in all these cases is that people are doing what was taught first in an introductory class because they simply don't know more advanced and appropriate methods.

A very interesting example are unit root tests in econometrics. While there are plenty of choices available to test against or for a unit root in the lag polynomial of a time series (e.g., the (Augmented) Dickey Fuller Test or the KPSS test), the problem can be circumvented completely when one uses Bayesian analysis. Sims pointed this out in his provocative paper titled Understanding Unit Rooters: A Helicopter Tour from 1991.

Unit root tests remain valid and used in econometrics. While I personally would attribute this mostly to people being reluctant to adjust to Bayesian practices, many conservative econometricians defend the practice of unit root tests by saying that a Bayesian view of the world contradicts the premise of econometric research. (That is, economists think of the world as a place with fixed parameters, not random parameters that are governed by some hyperparameter.)

• I would be interested in a brief discussion of how Bayesian practices circumvent these tests. In other words, how would you make the case for this claim? – Mike Hunter Jun 18 '16 at 21:06
• I have to admit that it has been a while since I read the paper, but the main point is that using a flat prior for the Bayesian analysis of a time series, one can use the standard t-values. – Jeremias K Jun 19 '16 at 8:25

Paying licensing fees for high-quality statistical software systems. #R

Teaching/conducting two-tailed tests for difference without simultaneously testing for equivalence in the frequentist realm of hypothesis testing is a deep commitment to confirmation bias.

There's some nuance, in that an appropriate power analysis with thoughtful definition of effect size can guard against this and provide more or less the same kinds of inferences, but (a) power analyses are so often ignored in presenting findings, and (b) I have never seen a power analysis for, for example, each coefficient estimated for each variable in a multiple regression, but it is straightforward to do so for combined tests for difference and tests for equivalence (i.e. relevance tests).

Using a Negative Binomial model rather than a (robust) Poisson model to identify a parameter of interest in a count variable, only because there is over-dispersion ?

The proof that Poisson is more robust in the case of fixed-effects is quite recent as it is offen made reference to: Wooldridge, J. M., “Distribution-Free Estimation of Some Nonlinear Panel Data Models,” Journal of Econometrics 90 (1999), 77–97.

Here are a few anachronisms:

• The neoplatonic assumption that there is a single, "true" population out there in the theoretical ether that is eternal, fixed and unmoving against which our imperfect samples can be evaluated does little to advance learning and knowledge.

• The reductionism inherent in mandates such as Occam's Razor is inconsistent with the times. OR can be summarized as, "Among competing hypotheses, the one with the fewest assumptions should be selected." Alternatives include Epicurus' Principle of Multiple Explanations, which roughly states, "If more than one theory is consistent with the data, keep them all."

• The whole peer-review system is desperately in need of an overhaul.

* Edit *

• With massive data containing tens of millions of features, there is no longer need for a variable selection phase.

• In addition, inferential statistics are meaningless.

• Comments are not for extended discussion; this conversation has been moved to chat. – whuber Jun 20 '16 at 13:47