What red flags should I be looking for when chi-squared independence test has p values of 0?

My expected and observed tables look pretty similar, so I wouldn't have expected to receive a p value of 0.0. I guess one potential problem could be that a few of my expected and received numbers are less than 5. Some are even less than 1.

Could this be causing the 0.0 p value, and if not, what else should I be looking for?

• What are the sizes of the residuals? Is the p-value truly 0 or are you quoting it rounded to the nearest 0.1? Are you confident you are running the calculation correctly? (For instance, do you get precisely the right result when you perform the same calculation on a dataset with a published answer?)
– whuber
Jul 14, 2017 at 19:31
• The p-value is truly 0.0000000000 for some of my categorical variables (not all). I performed the same calculation on a dataset with a published answer and got precisely the correct result. Jul 14, 2017 at 20:47
• Something I found on the internet... 'Use of the chi-square tests is inappropriate if any expected frequency is below 1 or if the expected frequency is less than 5 in more than 20% of your cells.' So I think I have a problem since I do have expected frequencies less than 1... IDK if I can increase my dataset to fix this, any suggestions? Like, yeah the accuracy is lower because of this problem that I have, but how much does this problem affect the accuracy? Jul 17, 2017 at 2:21
• The chi-squared test is perfectly valid for these data. Your rows aren't even remotely similar, as evidenced by the huge chi-squared residuals displayed. I am surprised the p value is as large as it is!
– whuber
Oct 5, 2020 at 18:15
• @whuber: The rows are in fact quite similar, use prop.table() to see ... Jun 7 at 4:10

The counts in your contingency table are quite large, and the chisquare test is perfectly valid, but it is also useful to remember that with large sample sizes (as here), even small deviations from independence can be detected. Using R:

your_table <- matrix(
c(92, 0, 15, 1, 13, 363, 954, 1, 0, 0,
23572, 405, 1220, 274, 6717, 89746, 306236, 1233, 207, 40),
byrow=TRUE, nrow=2)
rownames(your_table) <- c("Piggyback", "Primary")
colnames(your_table) <- c("Condo", "Coop", "Improved", "Mobile",
"Multifamily", "Pud", "Singlefamily", "Sitecondo", "Townhouse",
"Vacantland")

your_table
Condo Coop Improved Mobile Multifamily   Pud Singlefamily Sitecondo
Piggyback    92    0       15      1          13   363          954         1
Primary   23572  405     1220    274        6717 89746       306236      1233
Townhouse Vacantland
Piggyback         0          0
Primary         207         40


But for structure, look at proportions:

round(prop.table(your_table, 1), 3)
Condo  Coop Improved Mobile Multifamily   Pud Singlefamily Sitecondo
Piggyback 0.064 0.000    0.010  0.001       0.009 0.252        0.663     0.001
Primary   0.055 0.001    0.003  0.001       0.016 0.209        0.713     0.003
Townhouse Vacantland
Piggyback         0          0
Primary           0          0


It might be better to show the effects graphically. Let us show the odds for Piggyback against Primary:

(I will try to augment with a version also showing uncertainties)

If these differences between rows are important, depends on your use ... which you did not tell us! But, they are real:

chisq.test(your_table)

Pearson's Chi-squared test

data:  your_table
X-squared = 57.409, df = 9, p-value = 4.219e-09

Warning message:
In chisq.test(your_table) : Chi-squared approximation may be incorrect


The warning is because a few of the cells have very small (expected) counts, but in this case not a problem, check with simulation:

chisq.test(your_table, sim=TRUE,  B=1E6)

Pearson's Chi-squared test with simulated p-value (based on 1e+06
replicates)

data:  your_table
X-squared = 57.409, df = NA, p-value = 0.000407


For the record: Code for the plot:

odds <- function(col){ ### to calculate odds for each column
col[1] / col[2] }

ODDS <- apply(your_table,  2, FUN=odds)

k <- length(ODDS)

oddstab <- data.frame(ODDS, index=1:k)

library(ggplot2)
### The following plot is similar to a cleveland dot plot ...
ggplot(oddstab, aes(x=ODDS, y=index))  +
geom_text( aes(label=rownames(oddstab)), size=10  )  +
geom_segment( aes(x=0, xend=ODDS, y=index, yend=index), color="red" )  +
ylab(label=NULL)  +
theme(axis.ticks.y=element_blank(), axis.text.y=element_blank(),
plot.title=element_text(hjust=0.5),
plot.margin=margin(4, 39, 1, 12) )  +
ggtitle("Odds for Piggyback against Primary")  +
xlim( enlarge_lims(range(ODDS)))

enlarge_lims <- function( lims, factor=0.1 ) {
len <- diff(lims)
return( c(lims[1]-factor*len, lims[2] + factor*len) )
}