How to cope with exploratory data analysis and data dredging in small-sample studies?

Question

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited sample size and a lot of data gathered through different questionnaires (socio-demographics data, neuropsychological or medical scales -- e.g., mental or physical functioning, depression/anxiety level, symptoms checklist). It happens that EDA helps to highlight some unexpected relationships ("unexpected" meaning that they were not included in the initial analysis plan) that translates into additional questions/hypothesis.

As is the case for overfitting, data dredging or snooping does lead to results that do not generalize. However, when a lot of data is available, it is quite difficult (for the researcher or physician) to postulate a limited set of hypotheses.

I would like to know if there are well-acknowledged methods, recommendations, or rules of thumb that may help to delineate EDA in the case of small-sample studies.

Answer 1

I think the main thing is to be honest when reporting such results that they were unexpected findings from EDA and not part of the initial analysis plan based on an a priori hypothesis. Some people like to label such results 'hypothesis generating': e.g. the first hit from a search for this phrase on Google Scholar includes the following in the conclusion section of its abstract:

As this was an "exploratory" analysis, this effect should be considered as hypothesis generating and assessed prospectively in other trials...

Though note that although this was a post-hoc subgroup analysis it was from a randomized control trial, not an observational study, in which the problem gets worse. Philip Cole poured scorn on the idea that observational ('epidemiologic') studies can generate hypotheses in a deliberately provocative but entertaining commentary:

P Cole. The hypothesis generating machine. Epidemiology 1993; 4:271-273.

Answer 2

I just drop some references about data dredging and clinical studies for the interested reader. This is intended to extend @onestop's fine answer. I tried to avoid articles focusing only on multiple comparisons or design issues, although studies with multiple endpoints continue to present challenging and controversial discussions (long after Rothman's claims about useless adjustments, Epidemiology 1990, 1: 43-46; or see Feise's review in BMC Medical Research Methodology 2002, 2:8).

My understanding is that, although I talked about exploratory data analysis, my question more generally addresses the use of data mining, with its potential pitfalls, in parallel to hypothesis-driven testing.

1. Koh, HC and Tan, G (2005). Data Mining Applications in Healthcare. Journal of Healthcare Information Management, 19(2), 64-72.
2. Ioannidis, JPA (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
3. Anderson, DR, Link, WA, Johnson, DH, and Burnham, KP (2001). Suggestions for Presenting the Results of Data Analysis. The Journal of Wildlife Management, 65(3), 373-378. -- this echoes @onestop's comment about the fact that we have to acknowledge the data-driven exploration/modeling beyond the initial set of hypotheses
4. Michels, KB and Rosner, BA (1996). Data trawling: to fish or not to fish. Lancet, 348, 1152-1153.
5. Lord, SJ, Gebski, VJ, and Keech, AC (2004). Multiple analyses in clinical trials: sound science or data dredging?. The Medical Journal of Australia, 181(8), 452-454.
6. Smith, GD and Ebrahim, S (2002). Data dredging, bias, or confounding. BMJ, 325, 1437-1438.
7. Afshartous, D and Wolf, M (2007). Avoiding ‘data snooping’ in multilevel and mixed effects models. Journal of the Royal Statistical Society A, 170(4), 1035–1059
8. Anderson, DR, Burnham, KP, Gould, WR, and Cherry, S (2001). Concerns about finding effects that are actually spurious. Widlife Society Bulletin, 29(1), 311-316.