Setting the threshold p-value as part of hypothesis generation

Question

If I am trying to generate hypotheses, what are the drawbacks of setting my p-value cutoff after looking at the various p-values I get for my set of results?

For instance, if I initially did a large number of experiments looking at the effects of several thousand chemicals individually on cell growth and I find that they all have an effect that is significantly different from the control, does it make sense to set my p-value cutoff lower than 0.05? (I actually have several controls with different chemicals that are known to have little to no effect and these all show up as significantly different from each other also. What I want to do is set the cutoff as smaller than the lowest p-value of these controls, but I don't know if this is an acceptable way to use p-values.)

Furthermore, is it possible to have a sample size that is too large. If I repeated each experiment on several thousand cells, is that why I might get really low p-values for my results?

Answer 1

If you intend to use the Neyman-Pearson approach then you definitely cannot set the cutoff for significance after the data has been analysed. However, that is not the only approach to statistical inference, and in many cases it is not the best approach. N-P is certainly not well matched to a task that you specify as hypothesis generation.

N-P allows you to specify a maximally acceptable rate of false positive results, the alpha level that is most often unthinkingly set to 0.05. The N-P approach mostly deals with decisions about what to do next (significant, discard the null; not significant, accept the null) rather than dealing directly with the evidential meaning of the results.

Fisher's approach is incompatible with N-P and treats the data as evidence: it yields a p value that is an index of evidence against the null hypothesis. It is far more often compatible with the needs of scientific experimentation than the N-P approach, in my opinion, in so far as it allows the evidence from an experiment to be considered in light of any other information before any decision is made about what to do next. In contrast to the all-or-none results of an N-P analysis, it encourages experiments to be repeated or refined.

Specify the exact p values that you obtained from the experiment and interpret the results thoughtfully. If an interesting finding comes from the data rather than a pre-experiment hypothesis then the results should be taken as preliminary and, if sufficiently interesting, it may be worth repeating the experiment.

(You should note that it is fairly common to see statistical analyses and interpretations that are a hybrid of N-P and Fisher: the hybrid is always inappropriate.)

To answer your specific questions, I will do so (obliquely) as a pharmacologist: it is unlikely that all of thousands of chemicals will affect cell growth at low concentrations, but certain that all chemicals will do so at a high concentration. Paracelsus famously said (in Greek, I assume) "All drugs are poisons, dose determines effect." If your doses are large then it is not scientifically interesting to find that they are toxic. Perhaps you should test them at a wide range of concentrations (geometrical spacing of concentrations is efficient). The concentration at which a chemical has biological effects is at least as interesting as the magnitude of the effect, and much more interesting than the significance level obtained in an experiment. Make sure that you don't convert a biochemical and experimental design question into a question about statistical significance.

Answer 2

I suggest you try a different approach -- False Discovery Rate (FDR).

The FDR for any given P value cutoff is the expected fraction of those comparisons (with P less than your cutoff) where the null hypothesis is actually false (while 1.0 - FDR is the fraction of the comparisons where you expect the null hypothesis to be true). You call all comparisons with a P value less than your cutoff to be "a discovery" and the FDR is the fraction of those discoveries that are expected to be false (false positive findings).

You can either choose a FDR and find out what P value cutoff to use. Or you can compute FDR for any P value you choose.

Answer 3

I think you're really wondering about $p$-value correction. Bonferroni's is the simplest. You should use one if you have multiple post-hoc tests. This is what you were discussing, except that people typically consider this an adjustment to the $p$-value rather than an adjustment to $\alpha$.

Also, since your sample sizes are large, it's quite reasonable that you are getting low $p$-values. But if you still think that your sample size that is "too large", I would guess that you are violating an assumption of independence of observations. For instance, you would violate this assumption if you observe cells multiple times but do not account for how observations within the same cell are correlated.