0
$\begingroup$

I wanted to do a hypothesis testing for paired data. My null hypothesis is that Xi is 4 times as big as Yi. So i create a variable Zi = Xi - 4Yi.

The null hypothesis is that Zi = 0

I do the testing in R as follows:

t.value = (mean(Z) - 0) / (sd(Z) / sqrt(length(Z))) 
p.value = 2*pt(-abs(t.value), df=length(Z)-1)

I get a p-value higher than 0.05 and i fail to reject the null hypothesis.

My problem is, when i try different values for Z, for example Zi = Xi - 10Yi and Zi = Xi - 50Yi, i still fail to reject the null hypothesis which is strange to me.

My data looks normal but the values for Xi and Yi are tiny. (Xi mean =0.00245945, Yi mean = 0.00109909)

Is there an explanation for this, or am i doing something wrong?

Improve this question
$\endgroup$
3
  • $\begingroup$ Can $X_i$ and $Y_i$ be negative? What are their variances or standard deviations? $\endgroup$
    – Henry
    Commented Dec 5, 2017 at 9:02
  • $\begingroup$ Any particular reason why you compute the p-value manually? This doesn't seem to be a special case and therefore seems to be covered by the existing functions. $\endgroup$
    – David Ernst
    Commented Dec 5, 2017 at 9:15
  • $\begingroup$ Yes it can be, standard deviations of Xi = 0.008099045 and standard deviations of Yi = 0.009010012. $\endgroup$
    – G_z
    Commented Dec 5, 2017 at 9:15

1 Answer 1

Reset to default
1
$\begingroup$

If your n is tiny, your test could be so under-powered that it will fail to reject all but the most ridiculous null hypotheses.

edit: I see now that the standard deviations are 6 or 7 times as large as the difference of means. That makes for a quite small effect, Cohen's $d\approx 0.15$, if your sample is representative. If you construct a variable $Z$ that is different from $X-Y$, like the $X-4Y$ you had, this ratio will change accordingly. This still looks like a power problem. Here you see how low your power is with $n=45$:

power.t.test(45,0.14,0.85,0.05,NULL,"paired","two.sided")

 Paired t test power calculation 

          n = 45
      delta = 0.14
         sd = 0.85
  sig.level = 0.05
      power = 0.1896772
alternative = two.sided

NOTE: n is number of *pairs*, sd is std.dev. of *differences* within pairs

Here's how many observations you would need for halfway decent power of $0.8$. 

power.t.test(NULL,0.14,0.85,0.05,0.8,"paired","two.sided")

 Paired t test power calculation 

          n = 291.2539
      delta = 0.14
         sd = 0.85
  sig.level = 0.05
      power = 0.8
alternative = two.sided

NOTE: n is number of *pairs*, sd is std.dev. of *differences* within pairs

This is based on the assumption that the difference of means and the standard deviations (I didn't have the sd of $Z$, so I approximated it by the mean of the sds of $X$ and $Y$) as measured in your sample of $n=45$ are representative of the population. If you don't want to make those assumptions, you can also reason in terms of Cohen's $d$ and what would constitute a large enough effect to care about, that you would want to be able to detect.

Improve this answer
$\endgroup$
3
  • $\begingroup$ n = 45, what is the standard for a tiny n? $\endgroup$
    – G_z
    Commented Dec 5, 2017 at 9:18
  • $\begingroup$ Not that for a single t-test, more like 10. Try the built in t-test function just to be sure there isn't a mistake in your code. $\endgroup$
    – David Ernst
    Commented Dec 5, 2017 at 9:19
  • $\begingroup$ I tried t.test() and i got the same results. $\endgroup$
    – G_z
    Commented Dec 5, 2017 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.