0
$\begingroup$

Currently I have a dataset with an N of 3866 with two variables, Project (group variable, six levels) and Topbox (two levels; yes, no). We are looking to see if there is a difference in the proportion of 'Yes' responses between different projects (i.e., is there a significant difference between the project that has the lowest proportion of 'yes' responses versus the project that has the highest proportion of 'yes' responses). I've conducted a Chi-Square test in both SAS and R, and both give me different X^2 and p-values. Outputs shown below:

SAS:

proc freq data = aggregate; 
tables q57_topbox*project / chisq;
run;

Frequency output: Frequency output

Chi-square output: Chi-square output

R:

chisq.test(records_filter$q57_topbox, records_filter$project, 
           correct=FALSE)

Chi-square output:

    Pearson's Chi-squared test

data:  records_filter$q57_topbox and records_filter$project
X-squared = 21.216, df = 5, p-value = 0.0007373

I have a couple questions regarding this.

  1. The X^2 and the p-value are different and I am not sure why.
  2. Looking more into this, I'm not sure if a Chi-square statistic is the right test for such a dataset. The cell counts are not only widely variable (counts between 37 and 1143 in the 'Yes' response), but the cell counts are also bigger than what I'm seeing most people would use a Chi-square for. Therefore I'm unsure if this is the correct test to begin with.

Any help at all is appreciated, thank you so much for reading. Please let me know if there's any other info I can provide (I'm quite new to asking questions to stack exchange, so apologies!)

$\endgroup$
1
  • 2
    $\begingroup$ I am guessing the difference is in how the NA values are handled; judging from the degrees of freedom, SAS treats them as a valid category, but R drops them. Try removing them and rerun the SAS analysis to see what happens. $\endgroup$
    – jbowman
    Sep 8, 2023 at 18:09

2 Answers 2

3
$\begingroup$

The difference comes from SAS using "NA" as own category. If this is what you are after, then you can replace the NA in R by some explicit missing like "missing".

Regarding your second question: You could use an extension of Fisher's exact test for general two-way tables instead. But in practice, such considerations are often less important than they seem to be. Much more important are questions like "is my sample really randomly picked or is there some bias?" or "which hypothesis would I want to show?"

$\endgroup$
4
  • 1
    $\begingroup$ Ha ha! you answered as I was typing my comment! (+1) $\endgroup$
    – jbowman
    Sep 8, 2023 at 18:10
  • 1
    $\begingroup$ You typed faster than I did. Or maybe you started typing a few seconds before I did. In either case +1. And we mostly agree. But ... Fisher's exact test? Wouldn't the run time be huge with these data? $\endgroup$
    – Peter Flom
    Sep 8, 2023 at 18:12
  • 1
    $\begingroup$ @PeterFlom: Good point, I honestly did not try! $\endgroup$
    – Michael M
    Sep 8, 2023 at 18:19
  • 1
    $\begingroup$ Cannot believe I missed something like to exclude the NA's in SAS! Thank you so much for the answer, you're great! $\endgroup$ Sep 8, 2023 at 18:22
3
$\begingroup$

One thing is that the SAS program is including the NA (hence, df = 10) while R is not (hence, df = 5). As the proportion of NA varies hugely by project, SAS is given a much higher $\chi ^2$ and a much lower p.

I think that you probably want to exclude the NA, but it's really dependent on exactly what NA means, how it arose, and what your overall purpose is.

As for using chi-square: Neither the variation in cell sizes nor the cell sizes themselves are reasons not to use it. The real question is whether your question would be better served by a regression than by chi-square. There are a couple differences. One is that regression marks one variable as "depenendent" or "response" and the others as "independent" or "predictors". Chi-square treats the variables interchangeably.

$\endgroup$
1
  • 1
    $\begingroup$ Three people at the same time, neat! and +1 $\endgroup$
    – Michael M
    Sep 8, 2023 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.