# How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited sample size and a lot of data gathered through different questionnaires (socio-demographics data, neuropsychological or medical scales -- e.g., mental or physical functioning, depression/anxiety level, symptoms checklist). It happens that EDA helps to highlight some unexpected relationships ("unexpected" meaning that they were not included in the initial analysis plan) that translates into additional questions/hypothesis.

As is the case for overfitting, data dredging or snooping does lead to results that do not generalize. However, when a lot of data is available, it is quite difficult (for the researcher or physician) to postulate a limited set of hypotheses.

I would like to know if there are well-acknowledged methods, recommendations, or rules of thumb that may help to delineate EDA in the case of small-sample studies.

• I'm not quite sure why the size of your sample matters. Can you offer anymore specific reasoning as to why you think it is different for small n than it is for big n? – Andy W Oct 22 '10 at 3:39
• @Andy Because then it becomes very difficult to consider an holdout sample and/or class imbalance with very limited sample size ($13<n<25$) generally yields larger classification error rate when applying CV; some individuals might be considered as outliers when studying bivariate distributions; and measures gathered on instruments with their own measurement error are less reliable (small $n$, large $\sigma$). In a certain sense, it is sometimes difficult to disentangle an unexpected relationship from an artifact. – chl Oct 22 '10 at 5:29
• I think I can understand that sentiment if what your interested in is solely classification. I think for causal inference the problems with data snooping are the same (i.e. the problems aren't solved by increased power to identify relationships). I'll try to formulate this opinion into an answer. I may ask a question on the main forum in the meantime about the use of cross-validation for causal inference, as I have not come across any work in my field that does this. – Andy W Oct 22 '10 at 12:48
• @Andy Thanks. Hopefully, your question will receive a lot of interesting answers. – chl Oct 22 '10 at 13:40

I think the main thing is to be honest when reporting such results that they were unexpected findings from EDA and not part of the initial analysis plan based on an a priori hypothesis. Some people like to label such results 'hypothesis generating': e.g. the first hit from a search for this phrase on Google Scholar includes the following in the conclusion section of its abstract:

As this was an "exploratory" analysis, this effect should be considered as hypothesis generating and assessed prospectively in other trials...

Though note that although this was a post-hoc subgroup analysis it was from a randomized control trial, not an observational study, in which the problem gets worse. Philip Cole poured scorn on the idea that observational ('epidemiologic') studies can generate hypotheses in a deliberately provocative but entertaining commentary:

P Cole. The hypothesis generating machine. Epidemiology 1993; 4:271-273.

• +1 Thanks for the link (and the retag). I'll look into this direction. – chl Oct 2 '10 at 8:05

I just drop some references about data dredging and clinical studies for the interested reader. This is intended to extend @onestop's fine answer. I tried to avoid articles focusing only on multiple comparisons or design issues, although studies with multiple endpoints continue to present challenging and controversial discussions (long after Rothman's claims about useless adjustments, Epidemiology 1990, 1: 43-46; or see Feise's review in BMC Medical Research Methodology 2002, 2:8).

My understanding is that, although I talked about exploratory data analysis, my question more generally addresses the use of data mining, with its potential pitfalls, in parallel to hypothesis-driven testing.

1. Koh, HC and Tan, G (2005). Data Mining Applications in Healthcare. Journal of Healthcare Information Management, 19(2), 64-72.
2. Ioannidis, JPA (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
3. Anderson, DR, Link, WA, Johnson, DH, and Burnham, KP (2001). Suggestions for Presenting the Results of Data Analysis. The Journal of Wildlife Management, 65(3), 373-378. -- this echoes @onestop's comment about the fact that we have to acknowledge the data-driven exploration/modeling beyond the initial set of hypotheses
4. Michels, KB and Rosner, BA (1996). Data trawling: to fish or not to fish. Lancet, 348, 1152-1153.
5. Lord, SJ, Gebski, VJ, and Keech, AC (2004). Multiple analyses in clinical trials: sound science or data dredging?. The Medical Journal of Australia, 181(8), 452-454.
6. Smith, GD and Ebrahim, S (2002). Data dredging, bias, or confounding. BMJ, 325, 1437-1438.
7. Afshartous, D and Wolf, M (2007). Avoiding ‘data snooping’ in multilevel and mixed effects models. Journal of the Royal Statistical Society A, 170(4), 1035–1059
8. Anderson, DR, Burnham, KP, Gould, WR, and Cherry, S (2001). Concerns about finding effects that are actually spurious. Widlife Society Bulletin, 29(1), 311-316.
• This is just a recap' of what I read so far. Obviously, I will not accept my own answer. Any other thoughts would be much appreciated. – chl Oct 4 '10 at 17:58
• Thanks for accepting my answer chi, though your own reference list is much better and more recent. I really should have thought of a couple of them myself as I've got them on my hard drive, and may have even read parts of them... – onestop Oct 6 '10 at 8:18