# Rules of thumb for “modern” statistics

I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common pitfalls when interpreting results from experimental and observational studies and provide practical recommendations for statistical inference, or exploratory data analysis. But I feel that "modern" guidelines are somewhat lacking, especially with the ever growing use of computational and robust statistics in various fields, or the introduction of techniques from the machine learning community in, e.g. clinical biostatistics or genetic epidemiology.

Apart from computational tricks or common pitfalls in data visualization which could be addressed elsewhere, I would like to ask: What are the top rules of thumb you would recommend for efficient data analysis? (one rule per answer, please).

I am thinking of guidelines that you might provide to a colleague, a researcher without strong background in statistical modeling, or a student in intermediate to advanced course. This might pertain to various stages of data analysis, e.g. sampling strategies, feature selection or model building, model comparison, post-estimation, etc.

Don't forget to do some basic data checking before you start the analysis. In particular, look at a scatter plot of every variable you intend to analyse against ID number, date / time of data collection or similar. The eye can often pick up patterns that reveal problems when summary statistics don't show anything unusual. And if you're going to use a log or other transformation for analysis, also use it for the plot.

• I learnt this one the hard way. Twice. – onestop Nov 5 '10 at 16:38
• Yes! Look before you leap. Please, look at the data. – vqv Dec 19 '10 at 22:09
• Visual inspection of the data can inflate type I error if decisions are made post-hoc. I tend to run confirmatory analyses as they were prespecified and include results that were impacted by inspection as exploratory or sensitivity analyses. – AdamO Jun 17 '13 at 22:54

Keep your analysis reproducible. A reviewer or your boss or someone else will eventually ask you how exactly you arrived at your result - probably six months or more after you did the analysis. You will not remember how you cleaned the data, what analysis you did, why you chose the specific model you used... And reconstructing all this is a pain.

Corollary: use a scripting language of some kind, put comments in your analysis scripts, and keep them. What you use (R, SAS, Stata, whatever) is less important than having a completely reproducible script. Reject environments in which this is impossible or awkward.

• If you're going to use R, I'd recommend embedding your R code in an Sweave document that produces your report. That way the R code stays with the report. – John D. Cook Nov 2 '10 at 14:00

## There is no free lunch

A large part of statistical failures is created by clicking a big shiny button called "Calculate significance" without taking into account its burden of hidden assumptions.

## Repeat

Even if a single call to a random generator is involved, one may have luck or bad luck and so jump to the wrong conclusions.

Talk to the statistician before conducting the study. If possible, before applying for the grant. Help him/her understand the problem you are studying, get his/her input on how to analyze the data you are about to collect and think about what that means for your study design and data requirements. Perhaps the stats guy/gal suggests doing a hierarchical model to account for who diagnosed the patients - then you need to track who diagnosed whom. Sounds trivial, but it's far better to think about this before you collect data (and fail to collect something crucial) than afterwards.

On a related note: do a power analysis before starting. Nothing is as frustrating as not having budgeted for a sufficiently large sample size. In thinking about what effect size you are expecting, remember publication bias - the effect size you are going to find will probably be smaller than what you expected given the (biased) literature.

One thing I tell my students is to produce an appropriate graph for every p-value. e.g., a scatterplot if they test correlation, side-by-side boxplots if they do a one-way ANOVA, etc.

If you're deciding between two ways of analysing your data, try it both ways and see if it makes a difference.

This is useful in many contexts:

• To transform or not transform
• Non-parametric or parameteric test
• Spearman's or Pearson's correlation
• PCA or factor analysis
• Whether to use the arithmetic mean or a robust estimate of the mean
• Whether to include a covariate or not
• Whether to use list-wise deletion, pair-wise deletion, imputation, or some other method of missing values replacement

This shouldn't absolve one from thinking through the issue, but it at least gives a sense of the degree to which substantive findings are robust to the choice.

• Is it a quotation? I'm just wondering how trying alternative testing procedures (not analysis strategies!) may not somewhat break control of Type I error or initial Power calculation. I know SAS systematically returns results from parametric and non-parametric tests (at least in two-sample comparison of means and ANOVA), but I always find this intriguing: Shouldn't we decide before seeing the results what test ought to be applied? – chl Sep 18 '10 at 9:57
• @chl good point. I agree that the above rule of thumb can be used for the wrong reasons. I.e., trying things multiple ways and only reporting the result that gives the more pleasing answer. I see the rule of thumb as useful as a data analyst training tool in order to learn the effect of analysis decisions on substantive conclusions. I've seen many students get lost with decisions particularly where there is competing advice in the literature (e.g., to transform or not to transform) that often have minimal influence on the substantive conclusions. – Jeromy Anglim Sep 19 '10 at 5:45
• @chl no it's not a quotation. But I thought it was good to demarcate the rule of thumb from its rationale and caveats. I changed it to bold to make it clear. – Jeromy Anglim Sep 19 '10 at 10:11
• Ok, it makes sense to me to try different transformations and look if it provides a better way to account for the studied relationships; what I don't understand is to try different analysis strategies, although it is current practice (but not reported in published articles :-), esp. when they rely on different assumptions (in EFA vs. PCA, you assume an extra error term; in non-parametric vs. parametric testing, you throw away part of the assumptions, etc.). But, I agree the demarcation between exploratory and confirmatory analysis is not so clear... – chl Sep 19 '10 at 10:29
• This seems to me only useful for exploratory analysis or during training and validation steps. You will always need a final verifying testing step or otherwise you might fool yourself by certain significant results that work well once you got a desired difference according to your 'subjective' beliefs. Who is to judge which method works better? I personally, if I doubt different methods, then I test it on simulated data, in order to test such things as variance of estimators or robustness, etc. – Martijn Weterings Jun 29 '18 at 3:04

Question your data. In the modern era of cheap RAM, we often work on large amounts of data. One 'fat-finger' error or 'lost decimal place' can easily dominate an analysis. Without some basic sanity checking, (or plotting the data, as suggested by others here) one can waste a lot of time. This also suggests using some basic techniques for 'robustness' to outliers.

• Corollary: look whether someone coded a missing value as "9999" instead of "NA". If your software uses this value at face value, it will mess up your analysis. – Stephan Kolassa Nov 29 '12 at 14:12

Use software that shows the chain of programming logic from the raw data through to the final analyses/results. Avoid software like Excel where one user can make an undetectable error in one cell, that only manual checking will pick up.

• VisTrails is one system that helps this process. (I've used only homebrew systems; common group goals are more important than a particular tool.) – denis Jun 17 '11 at 10:10

Always ask yourself "what do these results mean and how will they be used?"

Usually the purpose of using statistics is to assist in making decisions under uncertainty. So it is important to have at the front of your mind "What decisions will be made as a result of this analysis and how will this analysis influence these decisions?" (e.g. publish an article, recommend a new method be used, provide $X in funding to Y, get more data, report an estimated quantity as E, etc.etc.....) If you don't feel that there is any decision to be made, then one wonders why you are doing the analysis in the first place (as it is quite expensive to do analysis). I think of statistics as a "nuisance" in that it is a means to an end, rather than an end itself. In my view we only quantify uncertainty so that we can use this to make decisions which account for this uncertainty in a precise way. I think this is one reason why keeping things simple is a good policy in general, because it is usually much easier to relate a simple solution to the real world (and hence to the environment in which the decision is being made) than the complex solution. It is also usually easier to understand the limitations of the simple answer. You then move to the more complex solutions when you understand the limitations of the simple solution, and how the complex one addresses them. • I agree with everything except on the notion to keep things simple. To me simplicity or complexity should be a function of the cost of improper decision that you eloquently explained. Simplicity can have negligible costs in one area (e.g. serving the wrong ad to a customer) and a wildly different cost in another (administering the wrong treatment to a patient). – Thomas Speidel Jun 26 '14 at 14:48 There can be a long list but to mention a few: (in no specific order) 1. P-value is NOT probability. Specifically, it is not the probability of committing Type I error. Similarly, CIs have no probabilistic interpretation for the given data. They are applicable for repeated experiments. 2. Problem related to variance dominate bias most the time in practice, so a biased estimate with small variance is better than an unbiased estimate with large variance (most of the time). 3. Model fitting is an iterative process. Before analyzing the data understand the source of data and possible models that fit or don't fit the description. Also, try model any design issues in your model. 4. Use the visualization tools, look at the data (for possible abnormalities, obvious trends etc etc to understand the data) before analyzing it. Use the visualization methods (if possible) to see how the model fits to that data. 5. Last but not the least, use statistical software for what they are made for (to make your task of computation easier), they are not a substitute for human thinking. • Your item 1 is incorrect: the P value is the probability of obtaining data as extreme, or more extreme, given the null hypothesis. As far as I know that means that P is a probability--conditional but a probability nonetheless. Your statement is correct in the circumstances that one is working within the Neyman-Pearson paradigm of errors, but not is one is working within the Fisherian paradigm where P values are idices of evidence against the null hypothesis. It is true that the paradigms are regularly mixed into an incoherent mish-mash, but both are 'correct' when used alone and intact. – Michael Lew Apr 10 '11 at 9:37 • For confidence intervals you are, again, correct only within the confines of Neymanian confidence intervals. Fisher (and others before him) also devised and used things that one would interpret as confidence intervals, and there is a perfectly valid interpretation of such intervals refering to the particular experiment yielding the interval. In my opinion, they far preferable to Neyman's. See my answer to the question Discrete functions: Confidence interval coverage? for more detail: stats.stackexchange.com/questions/8844/… – Michael Lew Apr 10 '11 at 9:39 • @Michael you are correct, but lets see: How many times is the Null correct? Or better: Can anyone prove if the null is correct? We can also have deep philosophical debates about this but that is not the point. In quality control repetitions make sense, but in science any good decision rule must condition data. – suncoolsu Apr 10 '11 at 16:51 • Fisher knew this (conditioning on the observed data and the remark about quality control is based on that). He produced many counter examples based on this. Bayesian have been fighting about this, lets say, for more than half-a-century. – suncoolsu Apr 10 '11 at 16:58 • @Michael Sorry if I wasn't clear enough. All I wanted to say: P-value is a probability ONLY when the null is true, but most of the times null is NOT true (as in: we never expect$\mu=0$to be true; we assume it to be true, but our assumption is practically incorrect.) In case you are interested, I can point out some literature discussing this idea in greater detail. – suncoolsu Apr 11 '11 at 15:36 For data organization/management, ensure that when you generate new variables in the dataset (for example, calculating body mass index from height and weight), the original variables are never deleted. A non-destructive approach is best from a reproducibility perspective. You never know when you might mis-enter a command and subsequently need to redo your variable generation. Without the original variables, you will lose a lot of time! Think hard about the underlying data generating process (DGP). If the model you want to use doesn't reflect the DGP, you need to find a new model. • How do you know, how can you know, what the DGP is. For example, I run time series in an area where I have yet to see well developed theory (why certain types of public spending occur). I don't think its possible to know the true process in this case. – user54285 Apr 12 at 22:23 For histograms, a good rule of thumb for number of bins in a histogram: square root of the number of data points Despite increasingly larger datasets and more powerful software, over-fitting models is a major danger to researchers, especially those who have not yet been burned by over-fitting. Over-fitting means that you have fitted something more complicated than your data and the state of the art. Like love or beauty, it is hard to define, let alone to define formally, but easier to recognise. A minimal rule of thumb is 10 data points for every parameter estimated for anything like classical regression, and watch out for the consequences if you ignore it. For other analyses, you usually need much more to do a good job, particularly if there are rare categories in the data. Even if you can fit a model easily, you should worry constantly about what it means and how far it is reproducible with even a very similar dataset. • That's generally seen as a rule of thumb for models where the response is conditionally normal. In other cases, it is too liberal. For example, for binary classification, the corresponding rule of thumb would be 15 observations in the less commonly occurring category for every variable; & for survival analysis, it would be 10 events (ie, not censored data) for every variable. – gung Aug 5 '16 at 15:12 • I agree. I'll edit, but why not post your own rule of thumb alongside with extended commentary. – Nick Cox Aug 5 '16 at 15:17 • You should highlight the last sentence "Even if you can fit a model easily, you should worry constantly about what it means and how far it is reproducible with even a very similar dataset." – Martijn Weterings Jun 29 '18 at 2:22 In a forecasting problem (i.e., when you need to forecast$Y_{t+h}$given$(Y_t,X_t)t>T$, with the use of a learning set$(Y_1,X_1),\dots, (Y_T,X_T)$), the rule of the thumb (to be done before any complex modelling) are 1. Climatology ($Y_{t+h}$forecast by the mean observed value over the learning set, possibly by removing obvious periodic patterns) 2. Persistence ($Y_{t+h}$forecast by the last observed value:$Y_t$). What I often do now as a last simple benchmark / rule of the thumb is using randomForest($Y_{t+h}$~$Y_t+X_t\$, data=learningSet) in R software. It gives you (with 2 lines of code in R) a first idea of what can be achieved without any modelling.

If the model won't converge easily and quickly, it could be the fault of the software. It is, however, much more common that your data are not suitable for the model or the model is not suitable for the data. It could be hard to tell which, and empiricists and theorists can have different views. But subject-matter thinking, really looking at the data, and constantly thinking about the interpretation of the model help as much as anything can. Above all else, try a simpler model if a complicated one won't converge.

There is no gain in forcing convergence or in declaring victory and taking results after many iterations but before your model really has converged. At best you fool yourself if you do that.

• "really looking at the data" it would be so nice when we get a NN that does this work for us. – Martijn Weterings Jun 29 '18 at 2:18
• It was called JWT. – Nick Cox Jun 29 '18 at 5:42

In instrumental variables regression always check the joint significance of your instruments. The Staiger-Stock rule of thumb says that an F-statistic of less than 10 is worrisome and indicates that your instruments might be weak, i.e. they are not sufficiently correlated with the endogenous variable. However, this does not automatically imply that an F above 10 guarantees strong instruments. Staiger and Stock (1997) have shown that instrumental variables techniques like 2SLS can be badly biased in "small" samples if the instruments are only weakly correlated with the endogenous variable. Their example was the study by Angrist and Krueger (1991) who had more than 300,000 observations - a disturbing fact about the notion of "small" samples.

• I have added the link to the article but I believe this answer stull needs some further formatting, I found it too difficult to emphasize the 'rule of thumb' based on scanning the article very quickly, and this answer is not very intuitive. – Martijn Weterings Jun 29 '18 at 2:36

There are no criteria to choose information criteria.

Once someone says something like "The ?IC indicates this, but it is known often to give the wrong results" (where ? is any letter you like), you know that you will have also to think about the model and particularly whether it makes scientific or practical sense.

No algebra can tell you that.

I read this somewhere (probably on cross validated) and I haven't been able to find it anywhere, so here goes...

## If you've discovered an interesting result, it's probably wrong.

It's very easy to get excited by the prospect of a staggering p-value or a near perfect cross validation error. I've personally ecstatically presented awesome (false) results to colleagues only to have to retract them. Most often, if it looks too good to be true...

'taint true. 'Taint true at all.

Try to be valiant rather than virtuous That is, don't let petty signs of non-Normality, non-independence or non-linearity etc. block your road if such indications need to be disregarded in order to have the data speak loud and clear. -- In Danish, 'dristig' vs. 'dydig' are the adjectives.

When analyzing longitudinal data be sure to check that variables are coded the same way in each time period.

While writing my dissertation, which entailed analysis of secondary data, there was a week or so of utter bafflement of a 1-unit shift in mean depression scores across an otherwise stable mean by year: it turned out that of one of the years in my data set, scale items for a validated instrument had been coded 1–4 instead of 0–3.

Your hypothesis should drive your choice of model, not the other way around.

To paraphrase Maslow, if you are a hammer, everything looks like a nail. Specific models come with blinders and assumptions about the world built right in: for example non-dynamic models choke on treatment-outcome feedback.

Use simulation to check where the structure of your model may be creating "results" which are simply mathematical artifacts of your model's assumptions

Perform your analysis on rerandomized variables, or on simulated variables known to be uncorrelated with one another. Do this many times and contrast averaged point estimates (and confidence or credible intervals) with the results you obtain on actual data: are they all that different?

I am a data analyst rather than a statistician but these are my suggestions.

1)Before you analyze data make sure the assumptions of your method are right. Once you see results they can be hard to forget even after you fix the problems and the results change.

2) It helps to know your data. I run time series and got a result that made little sense given recent years data. I reviewed the methods in light of that and discovered the averaging of models in the method was distorting results for one period (and a structural break had occurred).

3) Be careful about rules of thumb. They reflect the experiences of individual researchers from their own data and if their field is very different from yours their conclusions may not be correct for your data. Moreover, and this was a shock to me, statisticians often disagree on key points.

4) Try to analyze data with different methods and see if the results are similar. Understand that no method is perfect and be careful to check when you can for violations of the assumptions.