Rules of thumb for "modern" statistics

Question

I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common pitfalls when interpreting results from experimental and observational studies and provide practical recommendations for statistical inference, or exploratory data analysis. But I feel that "modern" guidelines are somewhat lacking, especially with the ever growing use of computational and robust statistics in various fields, or the introduction of techniques from the machine learning community in, e.g. clinical biostatistics or genetic epidemiology.

Apart from computational tricks or common pitfalls in data visualization which could be addressed elsewhere, I would like to ask: What are the top rules of thumb you would recommend for efficient data analysis? (one rule per answer, please).

I am thinking of guidelines that you might provide to a colleague, a researcher without strong background in statistical modeling, or a student in intermediate to advanced course. This might pertain to various stages of data analysis, e.g. sampling strategies, feature selection or model building, model comparison, post-estimation, etc.

Answer 1

Don't forget to do some basic data checking before you start the analysis. In particular, look at a scatter plot of every variable you intend to analyse against ID number, date / time of data collection or similar. The eye can often pick up patterns that reveal problems when summary statistics don't show anything unusual. And if you're going to use a log or other transformation for analysis, also use it for the plot.

Answer 2

Keep your analysis reproducible. A reviewer or your boss or someone else will eventually ask you how exactly you arrived at your result - probably six months or more after you did the analysis. You will not remember how you cleaned the data, what analysis you did, why you chose the specific model you used... And reconstructing all this is a pain.

Corollary: use a scripting language of some kind, put comments in your analysis scripts, and keep them. What you use (R, SAS, Stata, whatever) is less important than having a completely reproducible script. Reject environments in which this is impossible or awkward.

Answer 3

There is no free lunch

A large part of statistical failures is created by clicking a big shiny button called "Calculate significance" without taking into account its burden of hidden assumptions.

Repeat

Even if a single call to a random generator is involved, one may have luck or bad luck and so jump to the wrong conclusions.

Answer 4

One rule per answer ;-)

Talk to the statistician before conducting the study. If possible, before applying for the grant. Help him/her understand the problem you are studying, get his/her input on how to analyze the data you are about to collect and think about what that means for your study design and data requirements. Perhaps the stats guy/gal suggests doing a hierarchical model to account for who diagnosed the patients - then you need to track who diagnosed whom. Sounds trivial, but it's far better to think about this before you collect data (and fail to collect something crucial) than afterwards.

On a related note: do a power analysis before starting. Nothing is as frustrating as not having budgeted for a sufficiently large sample size. In thinking about what effect size you are expecting, remember publication bias - the effect size you are going to find will probably be smaller than what you expected given the (biased) literature.

Answer 5

One thing I tell my students is to produce an appropriate graph for every p-value. e.g., a scatterplot if they test correlation, side-by-side boxplots if they do a one-way ANOVA, etc.

Answer 6

If you're deciding between two ways of analysing your data, try it both ways and see if it makes a difference.

This is useful in many contexts:

• To transform or not transform
• Non-parametric or parameteric test
• Spearman's or Pearson's correlation
• PCA or factor analysis
• Whether to use the arithmetic mean or a robust estimate of the mean
• Whether to include a covariate or not
• Whether to use list-wise deletion, pair-wise deletion, imputation, or some other method of missing values replacement

This shouldn't absolve one from thinking through the issue, but it at least gives a sense of the degree to which substantive findings are robust to the choice.

Answer 7

Question your data. In the modern era of cheap RAM, we often work on large amounts of data. One 'fat-finger' error or 'lost decimal place' can easily dominate an analysis. Without some basic sanity checking, (or plotting the data, as suggested by others here) one can waste a lot of time. This also suggests using some basic techniques for 'robustness' to outliers.

Answer 8

Use software that shows the chain of programming logic from the raw data through to the final analyses/results. Avoid software like Excel where one user can make an undetectable error in one cell, that only manual checking will pick up.

Answer 9

Always ask yourself "what do these results mean and how will they be used?"

Usually the purpose of using statistics is to assist in making decisions under uncertainty. So it is important to have at the front of your mind "What decisions will be made as a result of this analysis and how will this analysis influence these decisions?" (e.g. publish an article, recommend a new method be used, provide $X in funding to Y, get more data, report an estimated quantity as E, etc.etc.....)

If you don't feel that there is any decision to be made, then one wonders why you are doing the analysis in the first place (as it is quite expensive to do analysis). I think of statistics as a "nuisance" in that it is a means to an end, rather than an end itself. In my view we only quantify uncertainty so that we can use this to make decisions which account for this uncertainty in a precise way.

I think this is one reason why keeping things simple is a good policy in general, because it is usually much easier to relate a simple solution to the real world (and hence to the environment in which the decision is being made) than the complex solution. It is also usually easier to understand the limitations of the simple answer. You then move to the more complex solutions when you understand the limitations of the simple solution, and how the complex one addresses them.

Answer 10

There can be a long list but to mention a few: (in no specific order)

1. P-value is NOT probability. Specifically, it is not the probability of committing Type I error. Similarly, CIs have no probabilistic interpretation for the given data. They are applicable for repeated experiments.

2. Problem related to variance dominate bias most the time in practice, so a biased estimate with small variance is better than an unbiased estimate with large variance (most of the time).

3. Model fitting is an iterative process. Before analyzing the data understand the source of data and possible models that fit or don't fit the description. Also, try model any design issues in your model.

4. Use the visualization tools, look at the data (for possible abnormalities, obvious trends etc etc to understand the data) before analyzing it. Use the visualization methods (if possible) to see how the model fits to that data.

5. Last but not the least, use statistical software for what they are made for (to make your task of computation easier), they are not a substitute for human thinking.

Answer 11

For data organization/management, ensure that when you generate new variables in the dataset (for example, calculating body mass index from height and weight), the original variables are never deleted. A non-destructive approach is best from a reproducibility perspective. You never know when you might mis-enter a command and subsequently need to redo your variable generation. Without the original variables, you will lose a lot of time!

Answer 12

Think hard about the underlying data generating process (DGP). If the model you want to use doesn't reflect the DGP, you need to find a new model.

Answer 13

For histograms, a good rule of thumb for number of bins in a histogram:

square root of the number of data points

Answer 14

Despite increasingly larger datasets and more powerful software, over-fitting models is a major danger to researchers, especially those who have not yet been burned by over-fitting. Over-fitting means that you have fitted something more complicated than your data and the state of the art. Like love or beauty, it is hard to define, let alone to define formally, but easier to recognise.

A minimal rule of thumb is 10 data points for every parameter estimated for anything like classical regression, and watch out for the consequences if you ignore it. For other analyses, you usually need much more to do a good job, particularly if there are rare categories in the data.

Even if you can fit a model easily, you should worry constantly about what it means and how far it is reproducible with even a very similar dataset.

Answer 15

In a forecasting problem (i.e., when you need to forecast $Y_{t+h}$ given $(Y_t,X_t)$ $t>T$, with the use of a learning set $(Y_1,X_1),\dots, (Y_T,X_T)$ ), the rule of the thumb (to be done before any complex modelling) are

1. Climatology ($Y_{t+h}$ forecast by the mean observed value over the learning set, possibly by removing obvious periodic patterns)
2. Persistence ($Y_{t+h}$ forecast by the last observed value: $Y_t$).

What I often do now as a last simple benchmark / rule of the thumb is using randomForest($Y_{t+h}$~$Y_t+X_t$, data=learningSet) in R software. It gives you (with 2 lines of code in R) a first idea of what can be achieved without any modelling.

Answer 16

If the model won't converge easily and quickly, it could be the fault of the software. It is, however, much more common that your data are not suitable for the model or the model is not suitable for the data. It could be hard to tell which, and empiricists and theorists can have different views. But subject-matter thinking, really looking at the data, and constantly thinking about the interpretation of the model help as much as anything can. Above all else, try a simpler model if a complicated one won't converge.

There is no gain in forcing convergence or in declaring victory and taking results after many iterations but before your model really has converged. At best you fool yourself if you do that.

Answer 17

In instrumental variables regression always check the joint significance of your instruments. The Staiger-Stock rule of thumb says that an F-statistic of less than 10 is worrisome and indicates that your instruments might be weak, i.e. they are not sufficiently correlated with the endogenous variable. However, this does not automatically imply that an F above 10 guarantees strong instruments. Staiger and Stock (1997) have shown that instrumental variables techniques like 2SLS can be badly biased in "small" samples if the instruments are only weakly correlated with the endogenous variable. Their example was the study by Angrist and Krueger (1991) who had more than 300,000 observations - a disturbing fact about the notion of "small" samples.

Answer 18

There are no criteria to choose information criteria.

Once someone says something like "The ?IC indicates this, but it is known often to give the wrong results" (where ? is any letter you like), you know that you will have also to think about the model and particularly whether it makes scientific or practical sense.

No algebra can tell you that.

Answer 19

I read this somewhere (probably on cross validated) and I haven't been able to find it anywhere, so here goes...

If you've discovered an interesting result, it's probably wrong.

It's very easy to get excited by the prospect of a staggering p-value or a near perfect cross validation error. I've personally ecstatically presented awesome (false) results to colleagues only to have to retract them. Most often, if it looks too good to be true...

'taint true. 'Taint true at all.

Answer 20

Try to be valiant rather than virtuous That is, don't let petty signs of non-Normality, non-independence or non-linearity etc. block your road if such indications need to be disregarded in order to have the data speak loud and clear. -- In Danish, 'dristig' vs. 'dydig' are the adjectives.

Answer 21

When analyzing longitudinal data be sure to check that variables are coded the same way in each time period.

While writing my dissertation, which entailed analysis of secondary data, there was a week or so of utter bafflement of a 1-unit shift in mean depression scores across an otherwise stable mean by year: it turned out that of one of the years in my data set, scale items for a validated instrument had been coded 1–4 instead of 0–3.

Answer 22

Your hypothesis should drive your choice of model, not the other way around.

To paraphrase Maslow, if you are a hammer, everything looks like a nail. Specific models come with blinders and assumptions about the world built right in: for example non-dynamic models choke on treatment-outcome feedback.

Answer 23

Use simulation to check where the structure of your model may be creating "results" which are simply mathematical artifacts of your model's assumptions

Perform your analysis on rerandomized variables, or on simulated variables known to be uncorrelated with one another. Do this many times and contrast averaged point estimates (and confidence or credible intervals) with the results you obtain on actual data: are they all that different?

Answer 24

I am a data analyst rather than a statistician but these are my suggestions.

1)Before you analyze data make sure the assumptions of your method are right. Once you see results they can be hard to forget even after you fix the problems and the results change.

2) It helps to know your data. I run time series and got a result that made little sense given recent years data. I reviewed the methods in light of that and discovered the averaging of models in the method was distorting results for one period (and a structural break had occurred).

3) Be careful about rules of thumb. They reflect the experiences of individual researchers from their own data and if their field is very different from yours their conclusions may not be correct for your data. Moreover, and this was a shock to me, statisticians often disagree on key points.

4) Try to analyze data with different methods and see if the results are similar. Understand that no method is perfect and be careful to check when you can for violations of the assumptions.