Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem.
On the other hand, "exploratory data analysis" seems to be a perfectly respectable procedure in statistics, at least judging by the fact that a book with that title is still reverentially cited as a classic.
In my line of work I often come across what looks to me like rampant "data snooping", or perhaps it would be better described as "data torture", though those doing it seem to see the same activity as entirely reasonable and unproblematic "exploration".
Here's the typical scenario: costly experiment gets carried out (without much thought given to the subsequent analysis), the original researchers cannot readily discern a "story" in the gathered data, someone gets brought in to apply some "statistical wizardry", and who, after slicing and dicing the data every which way, finally manages to extract some publishable "story" from it.
Of course, there's usually some "validation" thrown in the final report/paper to show that the statistical analysis is on the up-and-up, but the blatant publish-at-all-cost attitude behind it all leaves me doubtful.
Unfortunately, my limited understanding of the do's and don'ts of data analysis keeps me from going beyond such vague doubts, so my conservative response is to basically disregard such findings.
My hope is that not only a better understanding of the distinction between exploration and snooping/torturing, but also, and more importantly, a better grasp of principles and techniques for detecting when that line has been crossed, will allow me to evaluate such findings in a way that can reasonably accounts for a less-than-optimal analytic procedure, and thus be able to go beyond my current rather simple-minded response of blanket disbelief.
EDIT: Thank you all for the very interesting comments and answers. Judging by their content, I think I may have not explained my question well enough. I hope this update will clarify matters.
My question here concerns not so much what I should do to avoid torturing my data (although this is a question that also interests me), but rather: how should I regard (or evaluate) results that I know for a fact have been arrived through such "data torture."
The situation gets more interesting in those (much rarer) cases in which, in addition, I am in the position to voice an opinion on such "findings" before they get submitted for publication.
At this point the most I can do is say something like "I don't know how much credence I can give to these findings, given what I know about the assumptions and procedures that went into getting them." This is too vague to be worth even saying. Wanting to go beyond such vagueness was the motivation for my post.
To be fair, my doubts here are based on more than seemingly questionable statistical methods. In fact, I see the latter more as consequence of the deeper problem: a combination of a cavalier attitude towards experimental design coupled with a categorical commitment to publishing the results as they stand (i.e. without any further experiments). Of course, follow-up projects are always envisioned, but it is simply out-of-the-question that not a single paper will come out of, say, "a refrigerator filled with 100,000 samples."
Statistics comes into the picture only as a means towards fulfilling this supreme objective. The only justification for latching onto the statistics (secondary as they are in the whole scenario) is that a frontal challenge to the assumption of "publication-at-all-cost" is simply pointless.
In fact, I can think of only one effective response in such situations: to propose some statistical test (not requiring additional experimentation) that truly tests the quality of the analysis. But I just don't have the chops in statistics for it. My hope (naive in retrospect) was to find out what I could study that may enable me to come up with such tests...
As I write this it dawns on me that, if it doesn't already exist, the world could use one new sub-branch of statistics, devoted to techniques for detecting and exposing "data-torture". (Of course, I don't mean getting carried away by the "torture" metaphor: the issue is not "data-torture" per-se, but the spurious "findings" it can lead to .)