1
$\begingroup$

I have the results of a survey and I want to see if there are significative differences between age groups (young, middle and senior) regarding to an answer. For instance, in the table below, we see that 95 % of young people said "Yes" (268 surveyed people) vs 5 % who said no (13 people).

enter image description here

Usually I would run a Chi-square test to test for statistical significance but in this case I obviously cant because I only have 1 senior person who said "No" making up the 7 % (and I need at least 5 answers in each case for a Chi-square test).

My question is; can I still use the other results? I mean, we just established that we cannot test statistical significance between age groups for this answer. But does this mean that the other answers (the 95% etc...) are not significant on their own ? Can I still use the results for the Young people for instance?

I have in mind that I need at least 30 values because of normality ; is this for each cell (which means that I cant use the 13 answers of young people saying "No")? For each row (again, I can only use the "Yes"s because I only have 24 "No"s)? Or the whole sample (I have 403 answers in total), in which case every result is significant on their own (even the 7% for the only senior who said "no") ?

Thank you very much

$\endgroup$
5
  • 1
    $\begingroup$ The categorization of age into 3 groups is an invalid way to deal with age effects. You'll find no magic in moving just across one of those age boundaries, and tons of outcome heterogeneity within each wide age interval. $\endgroup$ Oct 18, 2021 at 14:06
  • 2
    $\begingroup$ Expanding on the point by @Frankharrell, if your range for "young" people is 18-35, there will be minimal difference between a "young" 35-year-old and a "middle" 36-year-old, though a "young" 18-year-old and "young 35-year-old will have differences. Your binning misses this. (More concretely, a colleague is exactly 50 and dislikes the COVID statistics about people in their 50s, because he's more like a 49-year-old than a 59-year old, yet his "people in their 50s" statistics are based on 59-year-olds, not 49-year-olds.) $\endgroup$
    – Dave
    Oct 18, 2021 at 14:22
  • $\begingroup$ This was just an example; my real data is not about ages, I just happen to have 3 groups, so I made up this example for simplicity. $\endgroup$
    – Siva Kg
    Oct 18, 2021 at 15:24
  • 1
    $\begingroup$ These groups are so similar that it doesn't matter that one cell has an expectation of less than $5.$ That rule of thumb only matters when there is a medium-size discrepancy between the observed and expected values in the low-expectation cell. Because your data give an expectation of 0.6 for the No-Seniors cell, you have no problem insofar as the calculation of the p-value goes. (The $\chi^2$ p-value is 14%, whereas it should be 11%.) The lack of significance could be due to losing information in the binning process, as @FrankHarrell points out. This isn't "invalid" per se--it's just weak. $\endgroup$
    – whuber
    Oct 18, 2021 at 15:33
  • 4
    $\begingroup$ So, we are foolish for taking your example as presented. It is always better to explain the essence of your real problem. $\endgroup$
    – Nick Cox
    Oct 18, 2021 at 18:19

1 Answer 1

2
$\begingroup$

Whether it makes sense to try to distinguish between the the three age groups has been discussed in Comments.

Whether or not age categories make sense, a traditional chi-squared test (based on counts, never percentages) gives a P-value of questionable accuracy because you have an extremely small expected count (less than $1)$ for Seniors saying No. [It is small expected counts that matter; looking at observed counts can be misleading.]

TBL = rbind(c(268,101,9), c(13,11,1))
TBL
     [,1] [,2] [,3]
[1,]  268  101    9
[2,]   13   11    1

chisq.test(TBL)

        Pearson's Chi-squared test

data:  TBL
X-squared = 4.2454, df = 2, p-value =  0.1375

Warning message:
In chisq.test(TBL) : 
  Chi-squared approximation may be incorrect

chisq.test(TBL)$exp  # EXPECTED COUNTS
         [,1]       [,2]      [,3]
[1,] 263.56824 105.052109 9.3796526
[2,]  17.43176   6.947891 0.6203474

As implemented in R, a more useful P-value for this test can be simulated. But even then the P-value is nowhere near significance at the 5% level:

chisq.test(TBL, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 4.2454, df = NA, p-value = 0.1119

Not even the extended version of Fisher's Exact Test (for a $2 \times 3$ table) implemented in R shows significance at the 5% level:

fisher.test(TBL)$p.val
[1] 0.09588885

Whether or not the age categories make sense for your purposes, it seems clear that the percentages of No opinions are too nearly the same for all three age groups to show anything worth reporting. (As in @whuber's Comment, these tests are not invalid; the evidence for differences among groups is very weak; significant at 10% or 12%, not 5%.)

If you want to know whether the percentage of Yes votes is significantly above half, the answer--for all subjects taken together--the answer is that the null hypothesis is strongly rejected in favor of a right-sided alternative. That is to say, for all 402 subjects 69% Yes is significantly above 50% Yes.

        Exact binomial test

data:  278 and 378 + 24
number of successes = 278, number of trials = 402, 
 p-value = 5.523e-15
alternative hypothesis: 
 true probability of success is greater than 0.5
95 percent confidence interval:
  0.651437 1.000000
sample estimates:
 probability of success 
              0.6915423 

Note: If you want to do similar binomial tests for the the three age groups separately, you could do so. However, in order to avoid 'false discovery' with multiple tests on the same data, you should test at the 1% level rather than the 5% level.

Perhaps results for Seniors are unpersuasive--because you have only ten of them--regardless of P-value. (That has nothing to do with the fact that $10 < 30,$ but with the fact that ten is just very small, and it will be hard to believe they are truly representative of any particular population. Certainly, readers would wonder how they were selected and from what population.)

Also, any reports should make it clear that the original intent was to see if there might be significant differences among age groups, which there are not.

$\endgroup$
2
  • 1
    $\begingroup$ For reasons explained in my comment to the question, it is misleading to characterize the small expected count as "disastrously small." BTW, your count of "7" was presented in the question as "9," which is why we get slightly different p-values. $\endgroup$
    – whuber
    Oct 18, 2021 at 21:50
  • $\begingroup$ Thanks. Typo and its consequences corrected. $\endgroup$
    – BruceET
    Oct 18, 2021 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.