What are the worst (commonly adopted) ideas/principles in statistics?

In my statistical teaching, I encounter some stubborn ideas/principles relating to statistics that have become popularised, yet seem to me to be misleading, or in some cases utterly without merit. I would like to solicit the views of others on this forum to see what are the worst (commonly adopted) ideas/principles in statistical analysis/inference. I am mostly interested in ideas that are not just novice errors; i.e., ideas that are accepted and practiced by some actual statisticians/data analysts. To allow efficient voting on these, please give only one bad principle per answer, but feel free to give multiple answers.

• I don't know whether I would call these ideas "silly". If they are commonly accepted, then their wrongness is obviously not obvious. How about just calling them "worst ideas", or "worst practices"? Jul 10, 2020 at 10:54
• Jul 10, 2020 at 13:50
• Jul 10, 2020 at 14:37
• ARIMA : A marvel of theoretical rigor and mathematical elegance that is almost useless for any realistic business time series (auto.arima and similar automated tools always p,d,q orders that were just as well handled by a more intuitive ES approach). Jul 11, 2020 at 0:37
• @SkanderH.: do you want to post your comment as an answer? Maybe add a pointer to how incredulous the statistics community was when it turned out that ARIMA was no good at forecasting in the first M competition, as summarized very nicely and entertainingly by Hyndman (2020, IJF). Jul 17, 2020 at 2:36

I'll present one novice error (in this answer) and perhaps one error committed by more seasoned people.

Very often, even on this website, I see people lamenting that their data are not normally distributed and so t-tests or linear regression are out of the question. Even stranger, I will see people try to rationalize their choice for linear regression because their covariates are normally distributed.

I don't have to tell you that regression assumptions are about the conditional distribution, not the marginal. My absolute favorite way to demonstrate this flaw in thinking is to essentially compute a t-test with linear regression as I do here.

• I saw this one on Data Science earlier today: datascience.stackexchange.com/q/77456/73930. It also seems like we get this on here about every week.
– Dave
Jul 10, 2020 at 4:05
• Novice here. Regarding this point, are you essentially saying to group the outcome variable by the levels of the predictor (if categorical like this) and then to see whether that resulting distribution is normal? In fact, I thought that even this was besides the point and that it was simply the residuals of the model after you fit it that are required to be normal? Jul 10, 2020 at 6:58
• @fffrost I wouldn't recommend assessing the normality assumption via grouping by categorical predictors, but that is the idea yes. The assumption of normality is essentially about the residuals, which explains why this assumption is often assessed via QQplots of the residuals. Jul 10, 2020 at 11:51

Post hoc power analysis

That is, using power analysis after a study has been completed rather than before, and in particular plugging in the observed effect size estimate, sample size, etc.

Some people have the intuition that post hoc power analysis could be informative because it could help explain why they attained a non-significant result. Specifically, they think maybe their failure to attain a significant result doesn't mean their theory is wrong... instead maybe it's just that the study didn't have a large enough sample size or an efficient enough design to detect the effect. So then a post hoc power analysis should indicate low power, and we can just blame it on low power, right?

The problem is that the post hoc power analysis does not actually add any new information. It is a simple transformation of the p-value you already computed. If you got a non-significant result, then it's a mathematical necessity that post hoc power will be low. And conversely, post hoc power is high when and only when the observed p-value is small. So post hoc power cannot possibly provide any support for the hopeful line of reasoning mentioned above.

Here's another way to think about the conceptual problem with these kinds of post hoc power (PHP) exercises -- the following passage is from this paper by Russ Lenth:

Note that the problem here is not the chronological issue of running a power analysis after the study is completed per se -- it is possible to run after-the-fact power analysis in a way that is informative and sensible by varying some of the observed statistics, for example to estimate what would have happened if you had run the study in a different way. The key problem with "post hoc power analysis" as defined in this post is in simply plugging in all of the observed statistics when doing the power analysis. The vast majority of the time that someone does this, the problem they are attempting to solve is better solved by just computing some sort of confidence interval around their observed effect size estimate. That is, if someone wants to argue that the reason they failed to reject the null is not because their theory is wrong but just because the design was highly sub-optimal, then a more statistically sound way to make that argument is to compute the confidence interval around their observed estimate and point out that while it does include 0, it also includes large effect size values -- basically the interval is too wide to conclude very much about the true effect size, and thus is not a very strong disconfirmation.

• Ooh, this is a good one. +1 Jul 10, 2020 at 4:48
• Small nitpick: I think the posthoc analysis is a reasonable idea (thinking hard about minimal clinical significance value, learning from the nuisance parameters, etc), but is often done in an extremely lazy, meaningless way. The point I want to make is the issue is more with people not taking the PHP seriously rather than the core idea itself. This is important, because if one uses this argument to replace PHP with something else, we need to chose something that will be used well by the masses, not just us. Alternatively, maybe we shouldn't replace it all, just remove from requirements. Jul 13, 2020 at 15:21
• @CliffAB I agree with what I believe you are saying. I added a couple caveats & clarifications that I think are in the spirit of your point Jul 13, 2020 at 16:26

The idea that because something is not statistically significant, it is not interesting and should be ignored.

• Commonly quoted as "The absence of evidence is not the evidence of absence" I believe. Jul 10, 2020 at 4:35
• that quote is also wrong - absence of evidence is evidence of absence, just not proof of absence Jul 10, 2020 at 10:54
• @probabilityislogic conditioned on the fact I collect the right data and analyze it correctly, then I agree. Jul 10, 2020 at 11:55
• @probabilityislogic Absence of evidence of an effect is insufficient to serve as evidence of absence of an effect (of at least some size). You can actually provide evidence of the latter, and it is not the same thing as the former. Jul 13, 2020 at 16:57
• Aka, is it a lack of significance or a lack of measurement?
– user234562
Jul 24, 2020 at 15:28

Removing Outliers

It seems that many individuals have the idea that they not only can, but should disregard data points that are some number of standard deviations away from the mean. Even when there is no reason to suspect that the observation is invalid, or any conscious justification for identifying/removing outliers, this strategy is often considered a staple of data preprocessing.

• +1. I would add removing data points in general when they are arbitrarily deemed not representative (e.g. below instrument detectable limit) Aug 31, 2020 at 20:17

Not addressing multiple hypothesis testing problems.

Just because you aren't performing a t.test on 1,000,000 genes doesn't mean you're safe from it. One example of a field it notably pops up is in studies that test an effect conditional on a previous effect being significant. Often in experiments the authors identify a significant effect of something, and then conditional on it being significant, then perform further tests to better understand it without adjusting for that procedural analysis approach. I recently read a paper specifically about the pervasiveness of this problem in experiments, Multiple hypothesis testing in experimental economics and it was quite a good read.

This seems like low hanging fruit, but stepwise regression is one error which I see pretty frequently even from some stats people. Even if you haven't read some of the very well-written answers on this site which address the approach and its flaws, I think if you just took a moment to understand what is happening (that you are essentially testing with the data that generated the hypothesis) it would be clear that step wise is a bad idea.

Edit: This answer refers to inference problems. Prediction is something different. In my own (limited) experiments, stepwise seems to perform on par with other methods in terms of RMSE.

• Gelman: statmodeling.stat.columbia.edu/2014/06/02/… (The link agrees with you.)
– Dave
Jul 10, 2020 at 4:13
• @Dave God I love that blog. Its like the XKCD of stats. There is always a post by Gelman I can use for any situation. Jul 10, 2020 at 4:15
• Your answer seems to suggest stepwise regression is problematic regardless of its applications, which I think is incorrect. It also hints at one particular problem with stepwise regression, namely, hypothesis testing and taking the results at face value. In this I do see a problem. Could you perhaps make your answer more specific? Jul 23, 2020 at 17:53
• @RichardHardy In what applications would stepwise be advisable? Stepwise takes an "all or nothing" approach to variable selection, where as sparse models like LASSO at least allow for the variable to partly contribute to the linear predictor albeit in a reduced capacity. I think that's a much better approach than allowing a variable to either contribute fully or not. Open to arguments to the contrary though. Jul 23, 2020 at 18:27
• @DemetriPananos, That is exactly my point. If LASSO is considered a good method and it is not generally superior to stepwise regression, that is a good enough reason to conclude stepwise regression is a good method (at least for some problems). That is why I suggest you make your answer more specific to highlight the misuses of stepwise regression in contrast to criticising stepwise regression per se. Jul 24, 2020 at 7:09

Regression towards the mean is a far more common problem than is often realised.

It is also one of those things that is actually quite simple but appears to be quite nebulous on closer inspection, and this is partly due to the narrow way that it is usually taught. Sometimes it is attributed entirely to measurement error, and that can be quite misleading. It is often "defined" in terms of extreme events - for example, if a variable is sampled and an extreme value observed, the next measurement tends to be less extreme. But this is also misleading because it implies that it is the same variable being measured. Not only may RTM arise where the subsequent measures are on different variables, but it may arise for measures that are not even repeated measures on the same subject. For example some people recognise RTM from the original "discovery" by Galton who realised that the children of tall parents also tend to be tall but less tall than their parents, while children of short parents also tend to be short but less short than their parents.

Fundamentally, RTM is a consequence of imperfect correlation between two variables. Hence, the question shouldn't be about when RTM occurs - it should be about when RTM doesn't occur. Often the impact may be small but sometimes it can lead to completely spurious conclusions. A very simple one is the observation of a "placebo effect" in clinical trials. Another more subtle one, but potentially much more damaging is the inference of " growth trajectories" in lifecourse studies where conditioning on the outcome has implicitly taken place.

You have a nice answer to one that I posted a few weeks ago.

Debunking wrong CLT statement

False claim: the central limit theorem says that the empirical distribution converges to a normal distribution.

As the answers to my question show, that claim is utterly preposterous (unless the population is normal), yet the answers also tell me that this is a common misconception.

• There is this overstatement problem. I've heard it from professors 'we have 100 students taking the class, so the clt says the grades are normally distributed' Jul 27, 2020 at 11:32
• And there is the understatement problem. 'I cannot do a t test, my data isn't normal'. Here the clt would often help more than people see Jul 27, 2020 at 11:33

ARIMA!!! - a marvel of theoretical rigor and mathematical elegance that is almost useless for any realistic business time series.

Ok, that is an exaggeration: ARIMA and similar models like GARCH are occasionally useful. But ARIMA is not nearly as general purpose a model as most people seem to think it is.

Most competent Data Scientists and ML Engineers who are generalists (in the sense that they don't specialize in time series forecasting or econometrics), as well as MBA types and people with solid general statistics backgrounds, will default to ARIMA as the baseline model for a time series forecasting problem. Most of the time they end up sticking with it. When they do evaluate it against other models, it is usually against more exotic entities like Deep Learning Models, XGBoost, etc...

On the other hand, most time series specialists, supply chain analysts, experienced demand forecasting analysts, etc...stay away from ARIMA. The accepted baseline model and the one that is still very hard to beat is Holt-Winters, or Triple Exponential Smoothing. See for example "Why the damped trend works" by E S Gardner Jr & E McKenzie. Beyond academic forecasting, many enterprise grade forecasting solutions in the demand forecasting and the supply chain space still use some variation of Holt-Winters. This isn't corporate inertia or bad design, it is simply the case that Holt-Winters or Damped Holt-Winters is still the best overall approach in terms of robustness and average overall accuracy.

A brief history lesson:

Some history might be useful here: Exponential Smoothing models, Simple ES, Holt's model, and Holt-Winters, were developed in the 50s. They proved to be very useful and pragmatic, but were completely "ad-hoc". They had no underlying statistical theory or first principles - they were more of a case of: How can we extrapolate time series into the future? Moving averages are a good first step, but we need to make the moving average more responsive to recent observations. Why don't we just add an $$\alpha$$ parameter that gives more importance to recent observation? - This was how simple exponential smoothing was invented. Holt and Holt-Winters were simply the same idea, but with the trend and seasonality split up and then estimated with their own weighted moving average models (hence the additional $$\beta$$ and $$\gamma$$ parameters). In fact, in the original formulations of ES, the parameters $$\alpha$$, $$\beta$$, and $$\gamma$$ were chosen manually based on their gut feeling and domain knowledge.

Even today, I occasionally have to respond to requests of the type "The sales for this particular product division are highly reactive, can you please override the automated model selection process and set $$\alpha$$ to 0.95 for us" (Ahhh - thinking to myself - why don't y'all set it to a naive forecast then??? But I am an engineer, so I can't say things like that to a business person).

Anyway, ARIMA, which was proposed in the 1970s, was in some ways a direct response to Exponential Smoothing models. While engineers loved ES models, statisticians were horrified by them. They yearned for a model that had at least some theoretical justification to it. And that is exactly what Box and Jenkins did when they came up with ARIMA models. Instead of the ad-hoc pragmatism of ES models, the ARIMA approach was built from the ground up using sound first principles and highly rigorous theoretical considerations.

And ARIMA models are indeed very elegant and theoretically compelling. Even if you don't ever deploy a single ARIMA model to production in your whole life, I still highly recommend that anyone interested in time series forecasting dedicate some time to fully grasping the theory behind how ARIMA works, because it will give a very good understanding of how time series behave in general.

But ARIMA never did well empirically, see here. Hyndman writes (and quotes others):

Many of the discussants seem to have been enamoured with ARIMA models. “It is amazing to me, however, that after all this exercise in identifying models, transforming and so on, that the autoregressive moving averages come out so badly. I wonder whether it might be partly due to the authors not using the backwards forecasting approach to obtain the initial errors”. — W.G. Gilchrist

“I find it hard to believe that Box-Jenkins, if properly applied, can actually be worse than so many of the simple methods”. — Chris Chatfield

At times, the discussion degenerated to questioning the competency of the authors: “Why do empirical studies sometimes give different answers? It may depend on the selected sample of time series, but I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins”. — Chris Chatfield

When ARIMA performs well, it does so only because the models selected are equivalent to Exponential Smoothing models (there is some overlap between the ARIMA family and the ES family for $$ARIMA(p,d,q)$$ with low values of $$p$$, $$d$$, and $$q$$ - see here and here for details).

I recall once working with a very smart business forecaster who had a strong statistics background and who was unhappy that our production system was using exponential smoothing, and wanted us to shift to ARIMA instead. So him and I worked together to test some ARIMA models. He shared with me that in his previous jobs, there was some informal wisdom around the fact that ARIMA models should never have values of $$p$$, $$d$$, or $$q$$ higher than 2. Ironically, this meant that the ARIMA models we were testing were all identical to or very close to ES models. It is not my colleague's fault though that he missed this irony. Most introductory graduate and MBA level material on time series modeling focus significantly or entirely on ARIMA and imply (even if they don't explicitly say so) that it is the end all be all of statistical forecasting. This is likely a holdover from the mind set that Hyndman referred to in the 70s, of academic forecasting experts being "enamored" with ARIMA. Additionally, the general framework that unifies ARIMA and ES models is a relatively recent development and isn't always covered in introductory texts, and is also significantly more involved mathematically than the basic formulations of both ARIMA and ES models (I have to confess I haven't completely wrapped my head around it yet myself).

Ok, why does ARIMA perform so poorly?
Several reasons, listed in no particular order of importance:

• ARIMA requires polynomial trends: Differencing is used to remove the trend from a time series in order to make it mean stationary, so that autoregressive models are applicable. See this previous post for details. Consider a time series $$Y(t)=L(t)+T(t)$$ with $$L$$ the level and $$T$$ the trend (most of what I am saying is applicable to seasonal time series as well, but for simplicity's sake I will stick to the case trend only). Removing the trend amounts to applying a transformation that will map $$T(t)$$ to a constant $$T=c$$. Intuitively, the differencing component of ARIMA is the discrete time equivalent of differentiation. That is, for a discrete time series $$Y$$ that has an equivalent continuous time series $$Y_c$$, setting $$d = 1$$ ($$Y_n'= Y_n - Y_{n-1}$$) is equivalent to calculating $$\frac{dY_c}{dt}$$ and setting $$d=2$$ is equivalent to $$\frac{d^2Y_c}{dt^2}$$ etc...now consider what type of continuous curves can be transformed into constants by successive differentiation? Only polynomials of the form $$T(t)=a_nt^n+a_{n-1}t^{n-1}...+a_1t+a_0$$ (only? It's been a while since I studied calculus...) - note that a linear trend is the special case where $$T(t)=a_1t+a_0$$. For all other curves, no number of successive differentiations will lead to a constant value (consider and exponential curve or a sine wave, etc...). Same thing for discrete time differencing: it only transfroms the series into a mean stationary one if the trend is polynomial. But how many real world time series will have a higher order ($$n>2$$) polynomial trend? Very few if any at all. Hence selecting an order $$d>2$$ is a recipe for overfitting (and manually selected ARIMA models do indeed overfit often). And for lower order trends,$$d=0,1,2$$, you're in exponential smoothing territory (again, see the equivalence table here).

• ARIMA models assume a very specific data generating process: Data generating process generally refers to the "true" model that describes our data if we were able to observe it directly without errors or noise. For example an $$ARIMA(2,0,0)$$ model can be written as $$Y_t = a_1Y_{t-1}+a_2Y_{t-2}+c+ \epsilon_t$$ with $$\epsilon_t$$ modeling the errors and noise and the true model being $$\hat{Y}_t = a_1\hat{Y}_{t-1}+a_2\hat{Y}_{t-2}+c$$ but very few business time series have such a "true model", e.g why would a sales demand signal or a DC capacity time series ever have a DGP that corresponds to $$\hat{Y}_t = a_1\hat{Y}_{t-1}+a_2\hat{Y}_{t-2}+c??$$ If we look a little bit deeper into the structure of ARIMA models, we realize that they are in fact very complex models. An ARIMA model first removes the trend and the seasonality, and then looks at the residuals and tries to model them as a linear regression against passed values (hence "auto"-regression) - this will only work if the residuals do indeed have some complex underlying deterministic process. But many (most) business time series barely have enough signal in them to properly capture the trend and the seasonality, let alone remove them and then find additional autoregressive structure in the residuals. Most univariate business time series data is either too noisy or too sparse for that. That is why Holt-Winters, and more recently Facebook Prophet are so popular: They do away with looking for any complex pattern in the residuals and just model them as a moving average or don't bother modeling them at all (in Prophet's case), and focus mainly on capturing the dynamics of the seasonality and the trend. In short, ARIMA models are actually pretty complex, and complexity often leads to overfitting.

• Sometimes autoregressive processes are justified. But because of stationarity requirements, ARIMA AR processes are very weird and counter intuitive: Let's try to look at what types of processes correspond in fact to an auto-regressive process - i.e. what time series would actually have an underlying DGP that corresponds to an $$AR(p)$$ model. This is possible for example with a cell population growth model, where each cell reproduces by dividing into to 2, and hence the population $$P(t_n)$$ could reasonably be approximated by $$P_n = 2P_{n-1}+\epsilon_t$$. Because here $$a=2$$ ($$>1$$), the process is not stationary and can't be modeled using ARIMA. Nor are most "natural" $$AR(p)$$ models that have a true model of the form $$\hat{Y}_t = a_1\hat{Y}_{t-1}+a_2\hat{Y}_{t-2}...+a_p\hat{Y}_{t-p}+c$$This is because of the stationarity requirement: In order for the mean $$c$$ to remain constant, there are very stringent requirements on the values of $$a_1,a_2,...,a_p$$ (see this previous post) to insure that $$\hat{Y}_t$$ never strays too far from the mean. Basically,$$a_1,a_2,...,a_p$$ have to sort of cancel each other out $$\sum_{j=1}^pa_j<1$$otherwise the model is not stationary (this is what all that stuff about unit roots and Z-transforms is about). This implication leads to very weird DGPs if we were to consider them as "true models" of a business time series: e.g. we have a sales time series or an electricity load time series, etc...what type of causal relationships would have to occur in order to insure that$$\sum_{j=1}^pa_j<1?$$ e.g. what type of economic or social process could ever lead to a situation where the detrended sales for 3 weeks ago are always equal to negative the sum of the sales from 2 weeks ago and the sales from last week? Such a process would be outlandish to say the least. To recap: While there are real world processes that can correspond to an autoregressive model, they are almost never stationary (if anyone can think of a counter example - that is a naturally occurring stationary AR(p) process, please share, I've been searching for one for a while). A stationary AR(p) process behaves in weird and counter intuitive ways (more or less oscillating around the mean) that make them very hard to fit to business time series data in a naturally explainable way.

Hyndman mentions this (using stronger words than mine) in the aforementioned paper:

This reveals a view commonly held (even today) that there is some single model that describes the data generating process, and that the job of a forecaster is to find it. This seems patently absurd to me — real data come from processes that are much more complicated, non-linear and non-stationary than any model we might dream up — and George Box himself famously dismissed it saying, “All models are wrong but some are useful”.

But what about the 'good' ARIMA tools?

At this point would point out to some modern tools and packages that use ARIMA and perform very well on most reasonable time series (not too noisy or too sparse), such as auto.arima() from the R Forecast package or BigQuery ARIMA. These tools in fact rely on sophisticated model selection procedures which do a pretty good job of ensuring that the $$p,d,q$$ orders selected are optimal (BigQuery ARIMA also uses far more sophisticated seasonality and trend modeling than the standard ARIMA and SARIMA models do). In other words, they are not your grandparent's ARIMA (nor the one taught in most introductory graduate texts...) and will usually generate models with low $$p,d,q$$ values anyway (after proper pre-processing of course). In fact now that I think of it, I don't recall ever using auto.arima() on a work related time series and getting $$p,d,q > 1$$, although I did get a value of $$q=3$$ once using auto.arima() on the Air Passengers time series.

Conclusion

Learn traditional ARIMA models in and out, but don't use them. Stick to state space models (ES incredibly sophisticated descendants) or use modern automated ARIMA model packages (which are very similar to state space models under the hood anyway).

• I do not think ARIMA belongs among the worst statistical practices. The quoted paper contains several more mentions of ARIMA, and, contrary to the ones you chose to cite, they are positive. What you identify appear to be some historical misuses of ARIMA and some unfounded optimism from early 1980s. The misuses are not the model's fault, and they are already obsolete among professionals, but of course there will be people lagging behind a couple (or more) decades repeating the old mistakes. Jul 23, 2020 at 18:16
• That the true process is different from the model is common to most statistical models, not only ARIMA. You can bash most of the models this way. The relevant point is, the model should be useful, while it does not have to be correct. The closing recommendation to use modern automated ARIMA model packages contradicts the criticism of the model itself. Perhaps the answer should be reformulated, directing the criticism against certain misuses of the model rather than the model itself? (I do not disagree with the whole answer nor do I find it useless, but I think it can be improved.) Jul 23, 2020 at 18:22
• @RichardHardy: I do not think that ARIMA is among the worst practices. However, the continued unthinking preference for ARIMA as a benchmark certainly is. I have seen countless RfIs (Requests for Information) where we are asked whether our forecasting system contains ARIMA, which is completely, utterly, obviously useless in our use cases (retail forecasting). Nevertheless, one of the first things people learn when they read about forecasting is ARIMA, and they keep on asking for it. Its prevalence is out of all proportion to its practical usefulness. Jul 24, 2020 at 14:16
• @StephanKolassa, your comment sounds reasonable. I wish the answer adopted a similar tone. However, the current formulation (especially the opening statements) appear too harsh and not nuanced enough. Jul 24, 2020 at 19:42
• Some counter-evidence from current times: a 2020 paper "Distributed ARIMA Models for Ultra-long Time Series" by Rob J. Hyndman and coauthors says that ARIMA models also frequently serve as the benchmark methods for model combination because of their excellent performance in time series forecasting (Wang et al., 2019; Montero-Manso et al., 2020). Jul 26, 2020 at 9:30

Equating a high $$R^2$$ with a "good model" (or equivalently, lamenting - or, in the case of referees of papers, criticizing - that $$R^2$$ is "too" low). More discussion is provided, e.g. here and here.

As should be universally appreciated, $$R^2$$ increases (more precisely, never decreases, see here) in the number of regressors in the model, and can hence always be made equal to 1 by including sufficiently many powers and interaction terms in the model (see the related illustration below). That is, of course, a very bad idea because the resulting model will strongly overfit and hence predict very poorly out of sample.

Also, when you regress something onto itself, $$R^2$$ will be 1 by construction (as residuals are zero), but you have of course learnt nothing. Yet, praising high $$R^2$$ in similar setups (e.g., this year's GDP as a function of last year's, which in view of growth rates of around 2% is more or less the same) is not uncommon.

Conversely, a regression with a small $$R^2$$ can be highly interesting when the effect that is responsible for that $$R^2$$ is one that you can actually act upon (i.e., is causalish).

# R^2 increases even if you regress on pure noise
n <- 15
regressors <- n-1 # enough, as we'll also fit a constant
y <- rnorm(n)
X <- matrix(rnorm(regressors*n),ncol=regressors)

collectionR2s <- rep(NA,regressors)
for (i in 1:regressors){
collectionR2s[i] <- summary(lm(y~X[,1:i]))$r.squared } plot(1:regressors,collectionR2s,col="purple",pch=19,type="b",lwd=2) abline(h=1, lty=2)  • To be fair, this is not a problem of the R-squared in specific but rather of performance measures evaluated on the training data. So it is exactly the same problem for MSE, MAE, Accuracy, ... Jul 29, 2021 at 6:26 Forgetting that bootstrapping requires special care when examining distributions of non-pivotal quantities (e.g., for estimating their confidence intervals), even though that has been known since the beginning. • What are examples of non-pivotal quantities for which this mistake is made? Mar 26, 2021 at 9:31 • @Johannes Efron, in the paper linked above, noted the issue with respect to the distribution of correlation coefficient estimates. See Section 8, Remarks B and E (pages 19-22). The plug-in estimator for the Shannon entropy is even more troublesome; see this page for example. Dangers of blindly bootstrapping to get confidence intervals are discussed here. – EdM Mar 26, 2021 at 15:21 "A complex model is better than a simple one". Or a variation thereof: "We need a model that can model nonlinearities." Especially often heard in forecasting. There is a strong preconception that a more complex model will forecast better than a simple one. That is very often not the case. • Someone could post as a misconception that simple linear models are always a good idea.... Jul 13, 2020 at 17:43 • +1 Most of the time people have no idea how to take into account the uncertainties in the fit parameters of their complicated model when doing predictions. If you pretend that your point estimates are 100% accurate, you get very nice-looking forecasts that don't mean anything. Even worse: more often than not there is no agreed-upon method for propagating the errors in the model parameters into the forecast. Only Bayesian methods seem to have sure-fire methods for that. Jul 17, 2020 at 9:33 Doing statistical inference with a - most certainly - biased convenience sample. (And then caring primarily about normality instead of addressing bias...) What does a p-value mean? ALERT TO NEWCOMERS: THIS QUOTE IS EXTREMELY FALSE “The probability that the null hypothesis is true, duh! Come on, Dave, you’re a professional statistician, and that’s Statistics 101.” I get the appeal of this one, and it would be really nice to have a simple measure of the probability of the null hypothesis, but no. • It seems a stretch to think of this as a "commonly adopted idea in statistics." This instead belongs in a pantheon of mistakes made by non-statisticians. – whuber Jul 10, 2020 at 12:40 • There is some discrepancy between the title of the question ("commonly adopted") and the body of the question ("ideas that are accepted and practiced by some actual statisticians/data analysts", emphasis mine). By the latter, lower, threshold, I think this surely is a valid example. There is of course also no clear definition of a statistician/data analyst, but research suggests that, at least, applied researchers with considerable training in statistics make this mistake. E.g., psicothema.com/PDF/4266.pdf. Jul 10, 2020 at 12:53 It's not purely statistics, but more statistical modeling in the large sense, but a very common misconception, that I have also heard in some University courses, is that Random Forests cannot overfit. Here is a question where they asked exactly this, and I tried explaining why this isn't true, and where this misconception comes from. Assuming that controlling for covariates is equivalent to eliminating their causal impact—this is false. The original example given by Pearl is that of qualifications, gender, and hiring. We hope that qualifications affect hiring, and want to know if gender does too. Gender can affect qualifications (unequal opportunity to education, for example). If an average man with a given education is more likely to be hired than an average woman who happens to have that same level of education, that is evidence of sexism, right? Wrong. The conclusion of sexism would only be justifiable if there were no confounders between Qualifications and Hiring. On the contrary, it may be that the women who happened to have the same level of education came from wealthy families, and the interviewer was biased against them for that reason. In other words, controlling for covariates can open back door paths. In many cases, controlling for is the best we can do, but when other back door paths are likely to exist, the evidence for causal conclusions should be considered weak. • Whilst this point has some merit (+1), when I have seen statisticians control for a covariate, they are almost always aware that this only controls for its impact through paths that don't go through the other explanatory variables, and so it does not eliminate causal impacts through other explanatory variables in the model. In short, I agree with your substantive point that the described belief is an error, but I rarely see the statistical community make this error. (Though perhaps that is just a matter of who you talk to!) – Ben Aug 12, 2020 at 10:19 • @ben It's not just a problem with "paths that go through other explanatory variables": it's about opening back doors that would otherwise not be open. In other words, controlling for covariates can paradoxically make the situation worse. Aug 12, 2020 at 17:57 In the medical community especially, and somewhat less often in psychology, the "change from baseline" is usually analyzed by modelling the change as a function of covariates. Doug Altman and Martin Bland have a really great paper on why this is probably not a good idea and argue that an ANVOCA (post measure ~ covariates + baseline) is better. Frank Harrell also does a really great job of compiling some hidden assumptions behind this approach. • Even ANCOVA is biased unless the groups are balanced with respect to the outcome, which is quite a common problem in observational studies. Jul 10, 2020 at 7:24 People often assume that the uniform prior is uninformative. This is usually false. • Yes! I wonder how many people realise that saying$X \sim U(0,1)$is equivalent to saying$-\log(X) \sim Exp(1)\$ Jul 24, 2020 at 13:07

Not realizing to what extent functional form assumptions and parametrizations are buying information in your analysis. In economics, you get these models that seem really interesting and give you a new way to potentially identify some effect of interest, but sometimes you read them and realize that without that last normality assumption that gave you point identification, the model identifies infinite bounds, and so the model really isn't actually giving you anything helpful.

When analysing change, that it is OK to create change scores (followup - baseline or a percent change from baseline) and then regress them on baseline. It's not (mathematical coupling). ANCOVA is often suggested as the best approach and it might be in the case of randomisation to groups, such as in clinical trials, but if the groups are unbalanced as if often the case in observational studies, ANCOVA can also be biased.

• Funny, this is the premise of one of the formulations of the extended Kalman filter. :) Jul 23, 2020 at 15:38
• In non-randomized trials, ANCOVA is often as source of the Lord's paradox showing no effect, if there was a real difference at baseline which diminished later: change scores will find this "negative" effect, but ANCOVA won't. And the opposite in randomized studies with no post-effect: the analysis of change may "react" to the differences at baseline, which are nothing but an artefact and must be ignored (due to randomization), while the ANCOVA on post-values will correctly show no effect. BTW, in may medical problems we don't care of the final values, only the changes from the starting point. May 25 at 0:03
• And yes, it's widespread. The adjustment for baseline in case of analysing the change scores is formally advised by both major regulatory agencies, the FDA and EMA (and some locals) - link below. However it's often said to do it conditionally, if the correlation with baseline will exceed 0.5. stats.stackexchange.com/questions/459279/… May 25 at 0:05

Using interaction (product) terms in regressions without using curvilinear (quadratic) terms.

A few years ago I've been thinking about it (after seeing a few papers (in economic/management fields) that were doing it), and I realized that if in the true model the outcome variable depends on the square of some or all the variables in the model, yet those are not included and instead an interaction is included in the examined model, the researcher may find that the interaction has an effect, while in fact it does not.

I then searched to see if there is an academic paper that addressed this, and I did find one (could be more, but that is what I found): https://psycnet.apa.org/fulltext/1998-04950-001.html

You might say that it is a novice mistake, and that a real statistician should know to first try to include all terms and interactions of a certain degree in the regression. But still, this specific mistake seems to be quite common in many fields that apply statistics, and the above linked article demonstrates the misleading results it may lead to.

I vote for "specification tests," e.g., White's test for heteroscedasticity, Hausman's tests, etc. These are common in econometrics and elsewhere, to the point where many people think they comprise the actual definition of the assumptions tested rather than a means to evaluate them. You would think the recent ASA statements on p-values would have dampened the enthusiasm for these methods. However, a Google scholar search for "Hausman test" turns up 17,200 results since 2019 and 8,300 since 2020; i.e., they are not fading away.

• Could you elaborate on your second sentence? I am not a native speaker, perhaps that makes it difficult for me to understand it. Jul 27, 2020 at 14:10
• I have seen Ph.D. students in econ-related fields talk as if the meaning of the assumption is that it is "true" if it passes the test, and "false" otherwise. That's what I meant by "many people think they comprise the actual definition." Jul 27, 2020 at 15:06
• So do you mean that passing a test merely indicates there is insufficient evidence against the null in the direction of a specific alternative in contrast to all the directions of all the possible alternatives? Jul 27, 2020 at 15:32
• Sure, p>.05 obviously does not imply that the null model is true. But perhaps more importantly, p<.05 does not imply that alternative model should be used (even if true). It all depends on size of the discrepancy. Simulation is a valuable tool for learning more. Jul 27, 2020 at 20:51

“Correlation does not imply causation.”

This is a true statement. Even if there is causation, it could be in the opposite direction of what is asserted.

What I have seen happen is that, when the correlation is inconvenient, people take this to mean that correlation precludes causation.

I don’t see professional statisticians making this mistake, but I have seen it happen when people use that phrase to sound quantitative and rigorous in their analysis, only to botch the meaning.

• Correlation precludes causation is an even worse misinterpretation. But even saying that true correlation (not a type I error) can happen without some kind of direct, indirect, common cause or inverse causation is wrong. You don't know much about this causation, but you do know that it is there somewhere Jul 27, 2020 at 11:47

Using statistical significance (usually at $$1\%$$, $$5\%$$ or $$10\%$$) of explanatory variables / regressors as a criterion in model building for explanatory or predictive purposes.

In explanatory modelling, both subject-matter and statistical validity are needed; see e.g. the probabilistic reduction approach to model building by Aris Spanos described in "Effects of model selection and misspecification testing on inference: Probabilistic Reduction approach (Aris Spanos)" and references therein. Statistical validity of parameter estimators amounts to the certain statistical assumptions being satisfied by the data. E.g. for OLS estimators in linear regression models, this is homoskedasticity and zero autocorrelation of errors, among other things. There are corresponding tests to be applied on model residuals to yield insight on whether the assumptions are violated in a particular way. There is no assumption that the explanatory variables be statistically significant, however. Yet many a practitioner applies statistical significance of individual regressors or groups thereof as a criterion of model validity in model building, just like they apply the diagnostic tests mentioned above. In my experience, this is a rather common practice, but it is unjustified and thus a bad idea.

In predictive modelling, variable selection on the basis of statistical significance may be sensible. If one aims to maximize out-of-sample likelihood, AIC-based feature selection implies a cutoff level corresponding to a $$p$$-value of around $$16\%$$. But the commonly used thresholds of $$1\%$$, $$5\%$$ and $$10\%$$ are suboptimal for most purposes. Hence, using statistical significance of explanatory variables at common levels of $$1\%$$, $$5\%$$ and $$10\%$$ as a selection criterion is a bad idea also in predictive model building.

The 'rule of thumb' that the standard deviation $$S$$ of a normal sample can be usefully approximated as sample range $$D$$ divided by $$4$$ (or $$5$$ or $$6).$$

The rule is typically "illustrated" by an example, contrived so the 'rule' gives a reasonable answer. In fact, the appropriate divisor depends crucially on sample size $$n.$$

n=100
set.seed(2020)
s = replicate(10^5, sd(rnorm(n)))
set.seed(2020)  # same samples again
d = replicate(10^5, diff(range(rnorm(n))))
mean(d/s)
[1] 5.029495
summary(d/s)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
3.581   4.678   4.984   5.029   5.330   7.756


For, $$n = 25,$$ dividing the range by $$4$$ works pretty well, and without great variation. For $$n = 100$$ and $$500,$$ respective denominators are on average $$5$$ and $$6,$$ but with widely decreasing precision for individual samples as sample size increases. A simulation in R for $$n=100$$ is shown above.

Note: The idea of approximating $$S$$ as $$D/c_n$$ is not completely useless: For $$n < 15,$$ dividing the range by some constant $$c_n$$ (different for each $$n)$$ works well enough that makers of control charts often use range divided by the appropriate constant to get $$S$$ for chart boundaries.

• (+1) Even worse is the adoption of the premise that you know the sample is from a normal distribution! Even a slight alteration in the kurtosis can blow out the extreme values so that the sample range is very large indeed! Unless there is an extremely good theoretic reason to believe a sample comes from a normal distribution, using the extreme points for an estimator, on the assumption of a normal distribution is playing with fire!
– Ben
Aug 12, 2020 at 10:22

The idea that because we have in mind an "average" result, that a sequence of data that is either below or above the average means that a particular result "is due".

The examples are things like rolling a die, where a large number of "no six" outcomes are observed - surely a six is due soon!

• That one sounds more like a novice mistake than a widely accepted misconception, right? Jul 10, 2020 at 7:29
• If your chance of a hit is 0.25 and your first three attempts are misses, it is incorrect to say that on the fourth you are "due a hit".
– T_M
Jul 10, 2020 at 11:22
• But generally it is more subtle and even goes into philosophical territory about flavours of probabilistic uncertainty. Suppose I flip a coin 9 times and get 9 heads and I don't show you the result. Now I ask you: "What is the probability that after the 10th flip, I will have flipped a sequence of 10 heads in a row?" Even though 9 flips are already done, YOU have little choice but to stick with 2^{-10}. Had I shown you the result, you'd have said 1/2.
– T_M
Jul 10, 2020 at 11:27

My favorite stats malpractice: permuting features instead of samples in a permutation test. In genomics, it's common to get a big list of differentially expressed, or differentially methylated, or differentially accessible genes (or similar). Often this is full of unfamiliar items, because nobody knows the literature on all 30k human genes, let alone transcript variants or non-coding regions. So, it's common to interpret these lists by using tools like Enrichr to test for overlap with databases of biological systems or previous experiments.

Most such analyses yield p-values assuming that features (genes or transcripts) are exchangeable under some null hypothesis. This null hypothesis is much more restrictive than it seems at first, and I've never seen a case where it's a) biologically realistic or b) defended by any sort of diagnostic.

(Fortunately, there are tools that don't make this mistake. Look up MAST or CAMERA.)

Post-selection inference, i.e. model building and doing inference on the same data set where the inference does not account for the model building stage.

Either: Given a data set and no predetermined model, a model is built based on the patterns found in the data set.
Or: Given a data set and a model, the model is often found to be inadequate. The model is adjusted based on the patterns in the data set.
Then: The model is used for inference such as null hypothesis significance testing.
The problem: The inference cannot be taken at face value as it is conditional on the data set due to the model-building stage. Unfortunately, this fact often gets neglected in practice.

Examining the t-test for each variable in a regression, but not the F-tests for multiple variables.

A common practice in many fields that apply statistics, is to use a regression with many covariates in order to determine the effect of the covariates on the outcome(s) of interest.

In these researches it is common to use t-test for each of the covariates in order to determine whether we can say that this variable has an effect on the outcome or not.

(I'm putting aside the issue of how to identify a causal relation ("effect") - for now let's assume that there are reasonable identification assumptions. Or alternatively, the researcher is interested only in finding correlation, I just find it easier to speak of an "effect")

It could be that there are two or more variables that are somewhat highly correlated, and as a result including them both in the regression will yield a high p-value in each of their t-tests, but examining their combined contribution to the model by using an F-test may conclude that these variables, or at least one of them, has a great contribution to the model.

Some researches do not check for this, and therefore may disregard some very important factors that affect the outcome variable, because they only use t-tests.

Calling type I assertion probability the "type I error rate" when it is neither a rate nor the probability of making an error. It is the probability of making an assertion of an effect when there is no effect.

Calling type I assertion probability the "false positive rate" when it is not the probability of a false positive result. It is the probability of making an assertion of an effect when any assertion of an effect is by definition wrong. The probability of a false + result is the probability that an effect is not there given the evidence was + for such a finding. The is a Bayesian posterior probability, not $$\alpha$$.

Thinking that controlling $$\alpha$$ has to do with limiting decision errors.

• "Making an assertion of an effect when there is no effect" sounds like making an error. Could you please clarify?
– Dave
Jul 23, 2020 at 18:54
• Making an error implies that a mistake is being made. The probability of making a mistake is the probability that an effect is not present when you assert that it is present. This is just one minus the Bayesian posterior probability that the effect is present given the data. Think of a smoke alarm. We want P(fire | alarm) to be high and P(no fire | no alarm) to be high. Type I probability would be P(alarm | no fire) which doesn't help us very much. Jul 23, 2020 at 18:58
• But an alarm when there is no fire is, quite literally, a false alarm, isn't it?
– Dave
Jul 23, 2020 at 19:06
• Yes but it's not the mistake or probability that you care about. You want to know P(alarm mistaken) = P(mistake | alarm) = P(no fire | alarm), not P(alarm | no fire). Jul 23, 2020 at 19:43
• Sure, you care about P(no fire | alarm went off) if you're woken up by a blaring alarm, but if you're Honeywell designing the smoke detector to a particular sensitivity, wouldn't your concern be the frequency with which the alarm goes off even though there is no fire (and the frequency with which the alarm fails to go off when there is a fire)?
– Dave
Jul 23, 2020 at 20:19