10
$\begingroup$

I am trying to find a solution to compare two "goodness-of-fit chi-square" tests. More precisely, I want to compare results from two independent experiments. In these experiments the authors used the goodness-of-fit chi-square to compare random guessing (expected frequencies) with observed frequencies. The two experiments got the same number of participants and experimental procedures are identical, only the stimuli changed. The two experiments results indicated a significant chi-square (exp. 1 : X²(18)=45; p<.0005 and exp. 2 : X²(18)=79; p<.0001).

Now, what I want to do is to test if there is a difference between these two results. I think a solution could be the use of confidence intervals but I don't know how to calculate these confidence intervals only with these results. Or maybe a test to compare effect size (Cohen's w)?

Anyone have a solution?

Thanks a lot!

F.D.

$\endgroup$
  • 1
    $\begingroup$ Hi Florian. Why not use a permutation test on the difference between the chi squares? $\endgroup$ – Tal Galili Feb 8 '12 at 16:55
  • $\begingroup$ Hi and thanks for your answer! Simply because I don't really know permutations tests. Is it possible to do permutation only with two chi-square values (I don't have raw data, only the results)? Thanks again :) $\endgroup$ – Florian Feb 8 '12 at 20:54
8
$\begingroup$

The very limited information you have is certainly a severe constraint! However, things aren't entirely hopeless.

Under the same assumptions that lead to the asymptotic $\chi^2$ distribution for the test statistic of the goodness-of-fit test of the same name, the test statistic under the alternative hypothesis has, asymptotically, a noncentral $\chi^2$ distribution. If we assume the two stimuli are a) significant, and b) have the same effect, the associated test statistics will have the same asymptotic noncentral $\chi^2$ distribution. We can use this to construct a test - basically, by estimating the noncentrality parameter $\lambda$ and seeing whether the test statistics are far in the tails of the noncentral $\chi^2(18, \hat{\lambda}) $ distribution. (That's not to say this test will have much power, though.)

We can estimate the noncentrality parameter given the two test statistics by taking their average and subtracting the degrees of freedom (a methods of moments estimator), giving an estimate of 44, or by maximum likelihood:

x <- c(45, 79)
n <- 18

ll <- function(ncp, n, x) sum(dchisq(x, n, ncp, log=TRUE))
foo <- optimize(ll, c(30,60), n=n, x=x, maximum=TRUE)
> foo$maximum
[1] 43.67619

Good agreement between our two estimates, not actually surprising given two data points and the 18 degrees of freedom. Now to calculate a p-value:

> pchisq(x, n, foo$maximum)
[1] 0.1190264 0.8798421

So our p-value is 0.12, not sufficient to reject the null hypothesis that the two stimuli are the same.

Does this test actually have (roughly) a 5% reject rate when the noncentrality parameters are the same? Does it have any power? We'll attempt to answer these questions by constructing a power curve as follows. First, we fix the average $\lambda$ at the estimated value of 43.68. The alternative distributions for the two test statistics will be noncentral $\chi^2$ with 18 degrees of freedom and noncentrality parameters $(\lambda-\delta, \lambda+\delta)$ for $\delta = 1, 2, \dots, 15$. We'll simulate 10000 draws from these two distributions for each $\delta$ and see how often our test rejects at, say, the 90% and 95% level of confidence.

nreject05 <- nreject10 <- rep(0,16)
delta <- 0:15
lambda <- foo$maximum
for (d in delta)
{
  for (i in 1:10000)
  {
    x <- rchisq(2, n, ncp=c(lambda+d,lambda-d))
    lhat <- optimize(ll, c(5,95), n=n, x=x, maximum=TRUE)$maximum
    pval <- pchisq(min(x), n, lhat)
    nreject05[d+1] <- nreject05[d+1] + (pval < 0.05)
    nreject10[d+1] <- nreject10[d+1] + (pval < 0.10)
  }
}
preject05 <- nreject05 / 10000
preject10 <- nreject10 / 10000

plot(preject05~delta, type='l', lty=1, lwd=2,
     ylim = c(0, 0.4),
     xlab = "1/2 difference between NCPs",
     ylab = "Simulated rejection rates",
     main = "")
lines(preject10~delta, type='l', lty=2, lwd=2)
legend("topleft",legend=c(expression(paste(alpha, " = 0.05")),
                          expression(paste(alpha, " = 0.10"))),
       lty=c(1,2), lwd=2)

which gives the following:

enter image description here

Looking at the true null hypothesis points (x-axis value = 0), we see that the test is conservative, in that it doesn't appear to reject as often as the level would indicate, but not overwhelmingly so. As we expected, it doesn't have much power, but it's better than nothing. I wonder if there are better tests out there, given the very limited amount of information you have available.

$\endgroup$
  • $\begingroup$ I'am newbie in this stuff, could I ask you how to run the script (if it was script) from jbowman answer. In my case, a try to get the OR from 90% CI. I'am really appreciate if one of you can explain it to me, and I use PASW17 $\endgroup$ – user9114 Feb 10 '12 at 18:50
  • $\begingroup$ Hello ash6. In fact it's a script for R software (for more info : r-project.org), not syntax for PASW17. So this script can be run directly in the R console. This script does not calculate confidence intervals but give you the p-value (here precisely > pchisq(x, n, foo$maximum ==> [1] p-value=0.1190264) corresponding to the test of a difference between the 2 experiments (here between two stimuli, in the case of alternative hypothesis), and here we can't reject the null hypothesis that the two experiments gave the same results. $\endgroup$ – Florian Feb 12 '12 at 21:02
3
$\begingroup$

You could get the Cramer's V, which is interpretable as a correlation, convert it to a Fisher's Z, and then the confidence interval of that is straightforward (SE = 1/sqrt(n-3): Z ± se * 1.96). After you get the ends of the CI you can convert them back to r.

Have you considered putting all of your counts into a contingency table with a further dimension of experiment?

$\endgroup$
  • $\begingroup$ I tought it wasn't possible to use a Phi with a Pearson goodness of fit chi-square (1 variable). That's why I talked about Cohen's w but formulas are really similars (phi=X²/n and w=sqrt(X²/n))! But if it's possible to calculate phi with this test and apply r to z transformation, would you agree to give us a reference to quote? We would like to use this test in an article and few reviewers can be very pickies with stats. It would be such a great help for us! About your question : we don't have raw data only X² value, df and p from a published article. Thanks a lot for your help! $\endgroup$ – Florian Feb 8 '12 at 20:52
  • $\begingroup$ Sorry... meant to put down Cramer's V, not phi. Cramer's V can be used like phi. $\endgroup$ – John Feb 8 '12 at 21:38
  • $\begingroup$ And no, I don't have a citation. If you have a large effect it won't matter if there is a little bias in this measure. If you don't have a large effect make sure you don't make big bones out of the "significance" of any test. $\endgroup$ – John Feb 8 '12 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.