# Significance disagrees between raw count z-score and percent change CI

I am estimating the difference between quantities of owned homes in particular census tracts between ACS 2021 5 yr estimates and ACS 2016 5 yr estimates. I'd like to know if the quantities of owned homes differ significantly between these two time periods. I am following ACS documentation in the 2018 ACS general handbook, primarily chapter 7 and chapter 8. Please note, while these link to the 2018 handbook, the formulas I reference match the analogous formulas in the 2020 handbook, but are conveniently available chapter by chapter.

When I produce an estimate of significance following formula (3) from chapter 7 to determine if the difference of the raw counts from the two time periods is significantly different from 0 at the 90% level, I find that the difference is not significantly different from 0. When I calculate the percent change and associated margin of error (from formulas (7) and (8) in chapter 8), I observe that the CI does not contain 0, so I believe the percent change to be significantly different from 0.

My primary question is this: How can the difference in counts be not significant, while the percent change is significant? Is this a case of trying to compare apples and oranges (and therefore significance may disagree) or am I incorrect in my process, causing erroneous disagreement?

Math below for reference:
Count2016 = 365, MOE2016 = 170, SE2016 = 103.48...
Count2021 = 193, MOE2021 = 104, SE2021 = 63.31...

Percent change CI calculation:
pct_change_est = ((193-365)/365) * 100 = -47.12
pct_change_moe = (1/365)sqrt(104^2 + (193/365)^2170^2)*100 = 37.66
This implies a 90% CI is made from -47.12 +/- 37.66, which will not cross 0 and is therefore significant.

Z-score calculation of raw difference:
abs((193-365)/sqrt(63.31^2+103.48^2)) = 1.42...
As 1.42 < 1.645, the critical value for the 90% level of the normal distribution, I conclude that the raw difference is not significantly different from 0.

Thank you for any insight and help you can provide!

• I'd expect that some people own homes for more than 5 years, which means that some of the home owners in 2016 and 2021 are the same. Of course I may not understand the data fully, but treating these as independent samples seems to be inappropriate to me. Commented May 25, 2023 at 13:28
• Thank you for this follow up @C
– Val
Commented Oct 12, 2023 at 13:02
• Thank you for this follow up @ChristianHennig! This is a good thought, as these tests are constructed for independent samples. The U.S. Census Bureau creates ACS 5 yr estimates by sampling just a subset of the population. Their documentation encourages making comparisons between non-overlapping intervals (as their official estimates are averaged over these intervals). Here, I use 2017-2021 averages and 2012-2016 averages. Perhaps it is possible than some homeowners were surveyed for the ACS in both time periods.
– Val
Commented Oct 12, 2023 at 13:09