0
$\begingroup$

Say I want to test the effect of two levels of my independent variable (IV) on some dependent variable (DV). For instance, administering a drug in a small vs large dosage, and then measuring reaction times for each. Assume also that I find that the effect of "high dosage" is greater than the effect of "small dosage".

Does the following argument have any statistical basis, or is it fallacious?

"To make such a result (the effect of this particular drug) more believable, studies are advised to use more than just two levels of the independent variable, e.g. a third intermediary level (medium drug dosage). This is because the finding high>low can happen spuriously with 1/2 probability, whereas the finding high>medium>low can only happen spuriously with 1/6 probability (since 3!=6). Thus, an A>B>C effect is more robust than merely A>B, in terms of proving the effect of the IV (with levels A,B,C) on some DV."

This argument seems sensible intuitively, from a frequentist definition of "chance level" and thus of a Type I error; but it also seems to me wrong, since if a high>low result is statistically significant, it is surely no less significant than the result high>medium>low, if the same alpha level has been used.

$\endgroup$
  • $\begingroup$ any more thoughts at all?.... $\endgroup$ – z8080 Nov 30 '17 at 9:56
1
$\begingroup$

This seems to make some kind of sense under very simple assumptions. If we take the null hypothesis to be that the drug has no effect then all orderings ought to occur with equal likelihood. Given that, if I observe A>B>C in $n$ out of $N$ trials I will be more confident (smaller p-value) than if I observe A>B in $n$ out of $N$ trials.

However, what seems overlooked is what is the likelihood of observing A>B or A>B>C under the alternative hypothesis? That is, how likely is it that I observe A>B (or A>B>C) in $n$ out of $N$ trials if the drug does have an effect. Also, suppose I observe A>C>B, does that imply no effect. That is, is A>B>C the only (3 value) outcome in favor of the alternative?

If one were to use a simple linear effects model, i.e. $y=\theta_0+\theta_1 c+\epsilon$, where $c$ is the drug concentration, then the test for $\theta_1>0$ would depend on the standard error of the slope, which might have little to no relation to the number of distinct values for $c$ measured.

$\endgroup$
  • $\begingroup$ Thanks, this is helpful in thinking about the problem! I would say that the situation I had in mind would assume that A>B>C is the only (3 value) outcome in favor of the alternative, yes... The linear effects model you describe seems to make my question trivial (which I have no trouble admitting it might be) - but I think this is because it involves continues changes, as opposed to categorical. $\endgroup$ – z8080 Nov 12 '17 at 15:04
  • $\begingroup$ any more thoughts at all?.... $\endgroup$ – z8080 Nov 30 '17 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.