1
$\begingroup$

My question concerns the method for constructing a confidence interval for a difference in means based on a randomization test (that can be found here , check under 'Confidence interval based on the randomization test'). I noticed the following problem with the code: when there is a non-significant difference in means (as is the case in the 'PlantGrowth' data Thompson uses on his webpage), his procedure produces a confidence interval that contains the difference in means from the observed data which is fine. However, I ran his code for several datasets with a significant difference in means (based on a simple t.test) and every time the code produced an interval that did not contain the observed difference in means.

Is there some sort of logical error in Thompson's reasoning or is it simply impossible to produce an interval that contains the observed difference if that difference is significant?

I tried to think about this and I came up with the following:

What Thompson's code basically does is:

  • Subtract a sequence of various 'nulhypotheses' from the treatment data.
  • For each nulhypothesis, the RT is executed and the p-value is determined by looking at the position of the observed teststatistic (from the original, unshifted data) in the randomization distribution (of the shifted data).
  • If the nulhypothesis cannot be rejected (i.e. p > significance level), that specific nulhypothesis value belongs to the confidence interval.
  • Doing these steps for the entire sequence of nulhypotheses yields a confidence interval.

Now, given this procedure, suppose that there is a significant difference between the baseline and treatment means of the observed data. This means that the p-value from an hypothesis test is smaller than the significance level and thus that the nullhypothesis of no difference is rejected.

If you then subtract a different nullhypothesis from the treatment data such that the difference in means becomes even larger, you will certainly get a significant difference and the nullhypothesis will be rejected so it will not be in included in the confidence interval.

So how can you then ever get a confidence interval that contains the observed difference in means? Is it impossible or am I missing something?

I hope I have stated my question clearly.

Thanks in advance for your time!

Regards,

Bart

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If I'm reading this right, the second step in the procedure you quote is incorrect. The p value should be obtained by computing the test statistic from the shifted data, and comparing it to a critical value.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. And I think you're right. Other publications I have read on the same topic also use the shifted test-statistic to obtain the p-value so I was already a bit wary of that step. However, I also tried computing the p-value from the shifted test statistic and, while it of course produces a different interval, the original problem (confidence interval does not contain observed statistic) remained... $\endgroup$
    – Bart
    Commented Dec 14, 2014 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.