My question concerns the method for constructing a confidence interval for a difference in means based on a randomization test (that can be found here , check under 'Confidence interval based on the randomization test'). I noticed the following problem with the code: when there is a non-significant difference in means (as is the case in the 'PlantGrowth' data Thompson uses on his webpage), his procedure produces a confidence interval that contains the difference in means from the observed data which is fine. However, I ran his code for several datasets with a significant difference in means (based on a simple t.test) and every time the code produced an interval that did not contain the observed difference in means.
Is there some sort of logical error in Thompson's reasoning or is it simply impossible to produce an interval that contains the observed difference if that difference is significant?
I tried to think about this and I came up with the following:
What Thompson's code basically does is:
- Subtract a sequence of various 'nulhypotheses' from the treatment data.
- For each nulhypothesis, the RT is executed and the p-value is determined by looking at the position of the observed teststatistic (from the original, unshifted data) in the randomization distribution (of the shifted data).
- If the nulhypothesis cannot be rejected (i.e. p > significance level), that specific nulhypothesis value belongs to the confidence interval.
- Doing these steps for the entire sequence of nulhypotheses yields a confidence interval.
Now, given this procedure, suppose that there is a significant difference between the baseline and treatment means of the observed data. This means that the p-value from an hypothesis test is smaller than the significance level and thus that the nullhypothesis of no difference is rejected.
If you then subtract a different nullhypothesis from the treatment data such that the difference in means becomes even larger, you will certainly get a significant difference and the nullhypothesis will be rejected so it will not be in included in the confidence interval.
So how can you then ever get a confidence interval that contains the observed difference in means? Is it impossible or am I missing something?
I hope I have stated my question clearly.
Thanks in advance for your time!