# What is the best way to analyze data if your experimental design changes while running the experiment?

I imagine this is a somewhat common situation in practice. I am thinking mainly in terms of preclinical drug trials.

1) During the course of the study a new technique is learned or some time/money is freed up to be able to use a new technique as a secondary outcome.

2) Some key piece of equipment breaks, the lab member with the skill to perform a technique leaves, or a planned technique is looking like it is yielding inconsistent results for unknown reasons so that outcome is dropped for financial reasons.

3) When time comes to analyze data it comes to the knowledge of the researcher that the planned methods are inadequate/inappropriate and they wish to use a different approach.

4) Experiment 1 (e.g. test drug A) does not look like it will pan out, so a financial decision is made to stop that experiment and instead test drug B. Alternatively some other part of the experiment may be modified such as cell culture or animal strain.

5) Study is stopped prematurely or additional funds are freed up to increase sample size.

I can think of more, but you get the idea. Are p values of any use under these circumstances? Should corrections for multiple comparisons be made in the case of #4? At what point does deviation from the plan invalidate the hypothesis testing procedure?

The best way to me would be to simply plot the data and describe it in the hopes it may be useful for someone. However this behaviour is discouraged in favor of performing significance/hypothesis tests and the data may not be published if these are not included.

EDIT: I was thinking about it and the actual best thing to do is to take the data and come up with a model to explain it that makes a precise prediction that can then be tested. It seems like this should always be the best thing to do, though.

This is a great question!

In my opinion the most important place to start an answer is to point out the distinction that should be drawn between P-values which are the result of a significance test and the decisions that are the results of hypothesis tests. Hypothesis tests are designed to control long term errors and are entirely frequentist in that the Frequentist or Repeated Sampling Principle is respected above all other considerations. In contrast the P-value from a significance test is an index of the evidence in the data and should be used for inference that is consistent with the Likelihood Principle.

Hypothesis tests and significance tests are usually assumed to be the same thing. Among the scientists with whom I hang out the most common test employed is a hybrid of the two which offers most of the disadvantages of both with few advantages. That hybrid has been written about many, including myself. See this paper for a description of the issues aimed at biomedical scientists: http://www.ncbi.nlm.nih.gov/pubmed/22394284.

The various scenarios with ad hoc alterations to experimental design that you present are very problematical to a frequentist analyst because the long run error rates associated with decisions depend on pre-specified criteria of $\alpha$ (usually 0.05), sample size etc. If the experiment changes between the design phase and implementation it is difficult to know what characteristics of design to impute. Thus the alterations in design affect the error rates and it is not always easy to make an appropriate adjustment to the analysis that would be acceptable to everyone.

For example, if an experiment is extended beyond the initial design sample size then it begs the question of whether the extension is an attempt to gain 'significance' from data that didn't quite make it at the designed end point. Such behaviour increases the risk of false positive results. So does stopping prematurely with a significant result.

If you view the data as evidence then the matching up of experimental design and implementation is much less important. A larger sample will provide more reliable and more convincing evidence. The P-value need not be 'adjusted' or 'corrected' for deviations from the planned sample size when it is viewed as an index of evidence.

Of course, what it means to be an index of evidence is not clear to most people. It turns out that for any P-value from a significance test of a certain sample size (pre-specified or not) there is a specific likelihood function. That likelihood function depicts the evidence that the P-value indexes. I have written a long paper on this topic that, by happy coincidence, I arXived just two days ago. Its title is "To P or not to P: on the evidential nature of P-values and their place in scientific inference" http://arxiv.org/abs/1311.0081. The next paragraph is a relevant extract:

The long-run error rates associated with an experiment are a property of the experimental design and the behaviour of the experimenter rather than of the data. The 'size' of the experiment, $\alpha$, is properly set before the data is available, so it cannot be data-dependent. In contrast, the P-value from a significance test is determined by the data rather than the arbitrary setting of a threshold. It cannot logically be an error rate because it doesn't force a decision in the way that inductive behaviour does, and if a decision is made to discard the null hypothesis when a small (presumably) P-value is observed, the decision is made on the basis of the smallness of the P-value in conjunction with whatever information that the experimenter considers relevant. Thus the rate of erroneous inferences is a function of not only the P-value but the quality and availability of additional information and, sometimes, the intuition of the experimenter. P-values are not error rates, whether 'observed', 'obtained' or 'implied'.

So, answers. Yes, P-values are still useful in your scenarios. Unadjusted P-values still index the likelihood functions that depict the evidence in the data and you can still make inferences using that evidence. You should not adjust the P-values for multiple comparisons, but instead understand that the evidence needs to be interpreted in light of the number of comparisons made. Deviations from the plan will invalidate frequentist hypothesis tests, but not necessarily invalidate a likelihood-based evaluation of the evidence.

Getting the most out of the evidence and avoiding inflated type I errors are not the same thing, and you cannot comply with the frequentist principle and the likelihood principle at the same time in many circumstances. Your edit makes it sound like you have a Bayesian model in mind. That is usually a good idea (and will usually comply with the likelihood principle).

• Thanks for the nice response. I have done a bit of reading on the Fisher vs Neyman-Pearson controversy and hybrid so I am aware of it. It sounds like you are arguing for Fisher's approach which I agree is much more palatable. One issue with the p-value as an "index of evidence" is I am not sure it is possible for a researcher to communicate all the specific circumstances of the experiment to others as would be required for an appropriate interpretation. Nov 2 '13 at 22:30
• One way to communicate all the specific circumstances of the experiment is to simply describe the actual circumstances of the experiment. The lamentable lack of experimental design detail and awful assumption that everything that was done was done with a preassigned design is a consequence of the pretence that the frequentist methods are appropriate to the scientific enterprise. Nov 2 '13 at 23:32
• Well lets take case #4 in my question. You think it would be appropriate for researchers to report all the studies they did that "did not pan out" in order to communicate the specific circumstances surrounding what they do publish? From your paper "a P-value obtained from an experiment represents the long-run frequency of obtaining data as extreme as the observed data, or more extreme, given that the null hypothesis is true." This is a good definition, and because the p-value is a long run frequency it can only really be interpreted in light of the number of p values calculated. Nov 3 '13 at 1:00
• ..continuing. It seems to me that if researchers fail to report everything they have tried, what will occur is a chronic overconfidence in the results. Yet it is impractical to report everything you have tried. Putting such a rule in place may actually end up discouraging pilot studies and exploratory analysis. Also I agree that the current situation with describing methods is horrible. It really could not get much worse. Nov 3 '13 at 1:06
• The paper I referred to is now available at arxiv.org/abs/1311.0081 Nov 4 '13 at 1:14