# Rules of thumb for “modern” statistics

I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common pitfalls when interpreting results from experimental and observational studies and provide practical recommendations for statistical inference, or exploratory data analysis. But I feel that "modern" guidelines are somewhat lacking, especially with the ever growing use of computational and robust statistics in various fields, or the introduction of techniques from the machine learning community in, e.g. clinical biostatistics or genetic epidemiology.

Apart from computational tricks or common pitfalls in data visualization which could be addressed elsewhere, I would like to ask: What are the top rules of thumb you would recommend for efficient data analysis? (one rule per answer, please)

I am thinking of guidelines that you might provide to a colleague, a researcher without strong background in statistical modeling, or a student in intermediary to advanced course. This might pertain to various stages of data analysis, e.g. sampling strategies, feature selection or model building, model comparison, post-estimation, etc.

-

Don't forget to do some basic data checking before you start the analysis. In particular, look at a scatter plot of every variable you intend to analyse against ID number, date / time of data collection or similar. The eye can often pick up patterns that reveal problems when summary statistics don't show anything unusual. And if you're going to use a log or other transformation for analysis, also use it for the plot.

-
I learnt this one the hard way. Twice. –  onestop Nov 5 '10 at 16:38
Yes! Look before you leap. Please, look at the data. –  vqv Dec 19 '10 at 22:09
Visual inspection of the data can inflate type I error if decisions are made post-hoc. I tend to run confirmatory analyses as they were prespecified and include results that were impacted by inspection as exploratory or sensitivity analyses. –  AdamO Jun 17 '13 at 22:54

Keep your analysis reproducible. A reviewer or your boss or someone else will eventually ask you how exactly you arrived at your result - probably six months or more after you did the analysis. You will not remember how you cleaned the data, what analysis you did, why you chose the specific model you used... And reconstructing all this is a pain.

Corollary: use a scripting language of some kind, put comments in your analysis scripts, and keep them. What you use (R, SAS, Stata, whatever) is less important than having a completely reproducible script. Reject environments in which this is impossible or awkward.

-
If you're going to use R, I'd recommend embedding your R code in an Sweave document that produces your report. That way the R code stays with the report. –  John D. Cook Nov 2 '10 at 14:00

## There is no free lunch

A large part of statistical failures is created by clicking a big shiny button called "Calculate significance" without taking into account its burden of hidden assumptions.

## Repeat

Even if a single call to a random generator is involved, one may have luck or bad luck and so jump to the wrong conclusions.

-

One thing I tell my students is to produce an appropriate graph for every p-value. e.g., a scatterplot if they test correlation, side-by-side boxplots if they do a one-way ANOVA, etc.

-

If you're deciding between two ways of analysing your data, try it both ways and see if it makes a difference.

This is useful in many contexts:

• To transform or not transform
• Non-parametric or parameteric test
• Spearman's or Pearson's correlation
• PCA or factor analysis
• Whether to use the arithmetic mean or a robust estimate of the mean
• Whether to include a covariate or not
• Whether to use list-wise deletion, pair-wise deletion, imputation, or some other method of missing values replacement

This shouldn't absolve one from thinking through the issue, but it at least gives a sense of the degree to which substantive findings are robust to the choice.

-
Is it a quotation? I'm just wondering how trying alternative testing procedures (not analysis strategies!) may not somewhat break control of Type I error or initial Power calculation. I know SAS systematically returns results from parametric and non-parametric tests (at least in two-sample comparison of means and ANOVA), but I always find this intriguing: Shouldn't we decide before seeing the results what test ought to be applied? –  chl Sep 18 '10 at 9:57
@chl good point. I agree that the above rule of thumb can be used for the wrong reasons. I.e., trying things multiple ways and only reporting the result that gives the more pleasing answer. I see the rule of thumb as useful as a data analyst training tool in order to learn the effect of analysis decisions on substantive conclusions. I've seen many students get lost with decisions particularly where there is competing advice in the literature (e.g., to transform or not to transform) that often have minimal influence on the substantive conclusions. –  Jeromy Anglim Sep 19 '10 at 5:45
@chl no it's not a quotation. But I thought it was good to demarcate the rule of thumb from its rationale and caveats. I changed it to bold to make it clear. –  Jeromy Anglim Sep 19 '10 at 10:11
Ok, it makes sense to me to try different transformations and look if it provides a better way to account for the studied relationships; what I don't understand is to try different analysis strategies, although it is current practice (but not reported in published articles :-), esp. when they rely on different assumptions (in EFA vs. PCA, you assume an extra error term; in non-parametric vs. parametric testing, you throw away part of the assumptions, etc.). But, I agree the demarcation between exploratory and confirmatory analysis is not so clear... –  chl Sep 19 '10 at 10:29

There can be a long list but to mention a few: (in no specific order)

1. P-value is NOT probability. Specifically, it is not the probability of committing Type I error. Similarly, CIs have no probabilistic interpretation for the given data. They are applicable for repeated experiments.

2. Problem related to variance dominate bias most the time in practice, so a biased estimate with small variance is better than an unbiased estimate with large variance (most of the time).

3. Model fitting is an iterative process. Before analyzing the data understand the source of data and possible models that fit or dont fit the description. Also, try model any design issues in your model.

4. Use the visualization tools, look at the data (for possible abnormalities, obvious trends etc etc to understand the data) before analyzing it. Use the visualization methods (if possible) to see how the model fits to that data.

5. Last but not the least, use statistical software for what they are made for (to make your task of computation easier), they are not a substitute for human thinking.

-
Your item 1 is incorrect: the P value is the probability of obtaining data as extreme, or more extreme, given the null hypothesis. As far as I know that means that P is a probability--conditional but a probability nonetheless. Your statement is correct in the circumstances that one is working within the Neyman-Pearson paradigm of errors, but not is one is working within the Fisherian paradigm where P values are idices of evidence against the null hypothesis. It is true that the paradigms are regularly mixed into an incoherent mish-mash, but both are 'correct' when used alone and intact. –  Michael Lew Apr 10 '11 at 9:37
For confidence intervals you are, again, correct only within the confines of Neymanian confidence intervals. Fisher (and others before him) also devised and used things that one would interpret as confidence intervals, and there is a perfectly valid interpretation of such intervals refering to the particular experiment yielding the interval. In my opinion, they far preferable to Neyman's. See my answer to the question Discrete functions: Confidence interval coverage? for more detail: stats.stackexchange.com/questions/8844/… –  Michael Lew Apr 10 '11 at 9:39
@Michael you are correct, but lets see: How many times is the Null correct? Or better: Can anyone prove if the null is correct? We can also have deep philosophical debates about this but that is not the point. In quality control repetitions make sense, but in science any good decision rule must condition data. –  suncoolsu Apr 10 '11 at 16:51
Fisher knew this (conditioning on the observed data and the remark about quality control is based on that). He produced many counter examples based on this. Bayesian have been fighting about this, lets say, for more than half-a-century. –  suncoolsu Apr 10 '11 at 16:58
@Michael Sorry if I wasn't clear enough. All I wanted to say: P-value is a probability ONLY when the null is true, but most of the times null is NOT true (as in: we never expect $\mu=0$ to be true; we assume it to be true, but our assumption is practically incorrect.) In case you are interested, I can point out some literature discussing this idea in greater detail. –  suncoolsu Apr 11 '11 at 15:36

Talk to the statistician before conducting the study. If possible, before applying for the grant. Help him/her understand the problem you are studying, get his/her input on how to analyze the data you are about to collect and think about what that means for your study design and data requirements. Perhaps the stats guy/gal suggests doing a hierarchical model to account for who diagnosed the patients - then you need to track who diagnosed whom. Sounds trivial, but it's far better to think about this before you collect data (and fail to collect something crucial) than afterwards.

On a related note: do a power analysis before starting. Nothing is as frustrating as not having budgeted for a sufficiently large sample size. In thinking about what effect size you are expecting, remember publication bias - the effect size you are going to find will probably be smaller than what you expected given the (biased) literature.

-

Use software that shows the chain of programming logic from the raw data through to the final analyses/results. Avoid software like Excel where one user can make an undetectable error in one cell, that only manual checking will pick up.

-
VisTrails is one system that helps this process. (I've used only homebrew systems; common group goals are more important than a particular tool.) –  denis Jun 17 '11 at 10:10

Question your data. In the modern era of cheap RAM, we often work on large amounts of data. One 'fat-finger' error or 'lost decimal place' can easily dominate an analysis. Without some basic sanity checking, (or plotting the data, as suggested by others here) one can waste a lot of time. This also suggests using some basic techniques for 'robustness' to outliers.

-
Corollary: look whether someone coded a missing value as "9999" instead of "NA". If your software uses this value at face value, it will mess up your analysis. –  Stephan Kolassa Nov 29 '12 at 14:12

Always ask yourself "what do these results mean and how will they be used?"

Usually the purpose of using statistics is to assist in making decisions under uncertainty. So it is important to have at the front of your mind "What decisions will be made as a result of this analysis and how will this analysis influence these decisions?" (e.g. publish an article, recommend a new method be used, provide $X in funding to Y, get more data, report an estimated quantity as E, etc.etc.....) If you don't feel that there is any decision to be made, then one wonders why you are doing the analysis in the first place (as it is quite expensive to do analysis). I think of statistics as a "nuisance" in that it is a means to an end, rather than an end itself. In my view we only quantify uncertainty so that we can use this to make decisions which account for this uncertainty in a precise way. I think this is one reason why keeping things simple is a good policy in general, because it is usually much easier to relate a simple solution to the real world (and hence to the environment in which the decision is being made) than the complex solution. It is also usually easier to understand the limitations of the simple answer. You then move to the more complex solutions when you understand the limitations of the simple solution, and how the complex one addresses them. - I agree with everything except on the notion to keep things simple. To me simplicity or complexity should be a function of the cost of improper decision that you eloquently explained. Simplicity can have negligible costs in one area (e.g. serving the wrong ad to a customer) and a wildly different cost in another (administering the wrong treatment to a patient). – Thomas Speidel Jun 26 '14 at 14:48 For data organization/management, ensure that when you generate new variables in the dataset (for example, calculating body mass index from height and weight), the original variables are never deleted. A non-destructive approach is best from a reproducibility perspective. You never know when you might mis-enter a command and subsequently need to redo your variable generation. Without the original variables, you will lose a lot of time! - For histograms, a good rule of thumb for number of bins in a histogram: square root of the number of data points - Think hard about the underlying data generating process. If the model you want to use doesn't reflect the DGP, you need to find a new model. - In a forecasting problem (i.e., when you need to forecast$Y_{t+h}$given$(Y_t,X_t)t>T$, with the use of a learning set$(Y_1,X_1),\dots, (Y_T,X_T)$), the rule of the thumb (to be done before any complex modelling) are 1. Climatology ($Y_{t+h}$forecast by the mean observed value over the learning set, possibly by removing obvious periodic patterns) 2. Persistance ($Y_{t+h}$forecast by the last observed value:$Y_t$). What I often do now as a last simple benchmark / rule of the thumb is using randomForest($Y_{t+h}$~$Y_t+X_t\$, data=learningSet) in R software. It gives you (with 2 lines of code in R) a first idea of what can be achieved without any modelling.

-

In instrumental variables regression always check the joint significance of your instruments. The Staiger-Stock rule of thumb says that an F-statistic of less than 10 is worrisome and indicates that your instruments might be weak, i.e. they are not sufficiently correlated with the endogenous variable. However, this does not automatically imply that an F above 10 guarantees strong instruments. Staiger and Stock (1997) have shown that instrumental variables techniques like 2SLS can be badly biased in "small" samples if the instruments are only weakly correlated with the endogenous variable. Their example was the study by Angrist and Krueger (1991) who had more than 300,000 observations - a disturbing fact about the notion of "small" samples.

-