There is a distinction which sometimes doesn't get enough attention, namely hypothesis generation vs. hypothesis testing, or exploratory analysis vs. hypothesis testing. You are allowed all the dirty tricks in the world to come up with your idea / hypothesis. But when you later test it, you must ruthlessly kill your darlings.

I'm a biologist working with high throughput data all the time, and yes, I do this "slicing and dicing" quite often. Most of the cases the experiment performed was not carefully designed; or maybe those who planned it did not account for all possible results. Or the general attitude when planning was "let's see what's in there". We end up with expensive, valuable and in themselves interesting data sets that I then turn around and around to come up with a story.

But then, it is only a story (possible bedtime). After you have selected a couple of interesting angles -- and here is the crucial point -- you must test it not only with independent data sets or independent samples, but preferably with an independent approach, an independent experimental system.

So I think such data torturing is fine, as long as you keep this distinction in mind and remember what you are doing, at what stage of scientific process you are. You can use moon phases or redefine 2+2 as long as you have an independent validation of the data. To put it on a picture:

Unfortunately, there are those who order a microarray to round up a paper after several experiments have been done and no story emerged, with the hope that the high throughput analysis shows something. Or they are confused about the whole hypothesis testing vs. generation thing.

There is a distinction which sometimes doesn't get enough attention, namely hypothesis generation vs. hypothesis testing, or exploratory analysis vs. hypothesis testing. You are allowed all the dirty tricks in the world to come up with your idea / hypothesis. But when you later test it, you must ruthlessly kill your darlings.

I'm a biologist working with high throughput data all the time, and yes, I do this "slicing and dicing" quite often. Most of the cases the experiment performed was not carefully designed; or maybe those who planned it did not account for all possible results. Or the general attitude when planning was "let's see what's in there". We end up with expensive, valuable and in themselves interesting data sets that I then turn around and around to come up with a story.

But then, it is only a story (possible bedtime). After you have selected a couple of interesting angles -- and here is the crucial point -- you must test it not only with independent data sets or independent samples, but preferably with an independent approach, an independent experimental system.

The importance of this last thing -- an independent experimental setup, not only independent set of measurements or samples -- is often underestimated. However, when we test 30,000 variables for significant difference, it often happens that while similar (but different) samples from the same cohort and analysed with the same method will not reject the hypothesis we based on the previous set. But then we turn to another type of experiment and another cohort, and our findings turn out to be the result of a methodological bias or are limited in their applicability.

That is why we often need several papers by several independent researchers to really accept a hypothesis or a model.

So I think such data torturing is fine, as long as you keep this distinction in mind and remember what you are doing, at what stage of scientific process you are. You can use moon phases or redefine 2+2 as long as you have an independent validation of the data. To put it on a picture:

Unfortunately, there are those who order a microarray to round up a paper after several experiments have been done and no story emerged, with the hope that the high throughput analysis shows something. Or they are confused about the whole hypothesis testing vs. generation thing.