0
$\begingroup$

I am estimating the difference between quantities of owned homes in particular census tracts between ACS 2021 5 yr estimates and ACS 2016 5 yr estimates. I'd like to know if the quantities of owned homes differ significantly between these two time periods. I am following ACS documentation in the 2018 ACS general handbook, primarily chapter 7 and chapter 8. Please note, while these link to the 2018 handbook, the formulas I reference match the analogous formulas in the 2020 handbook, but are conveniently available chapter by chapter.

When I produce an estimate of significance following formula (3) from chapter 7 to determine if the difference of the raw counts from the two time periods is significantly different from 0 at the 90% level, I find that the difference is not significantly different from 0. When I calculate the percent change and associated margin of error (from formulas (7) and (8) in chapter 8), I observe that the CI does not contain 0, so I believe the percent change to be significantly different from 0.

My primary question is this: How can the difference in counts be not significant, while the percent change is significant? Is this a case of trying to compare apples and oranges (and therefore significance may disagree) or am I incorrect in my process, causing erroneous disagreement?

Math below for reference:
Count2016 = 365, MOE2016 = 170, SE2016 = 103.48...
Count2021 = 193, MOE2021 = 104, SE2021 = 63.31...

Percent change CI calculation:
pct_change_est = ((193-365)/365) * 100 = -47.12
pct_change_moe = (1/365)sqrt(104^2 + (193/365)^2170^2)*100 = 37.66
This implies a 90% CI is made from -47.12 +/- 37.66, which will not cross 0 and is therefore significant.

Z-score calculation of raw difference:
abs((193-365)/sqrt(63.31^2+103.48^2)) = 1.42...
As 1.42 < 1.645, the critical value for the 90% level of the normal distribution, I conclude that the raw difference is not significantly different from 0.

Thank you for any insight and help you can provide!

Improve this question
$\endgroup$
3
  • $\begingroup$ I'd expect that some people own homes for more than 5 years, which means that some of the home owners in 2016 and 2021 are the same. Of course I may not understand the data fully, but treating these as independent samples seems to be inappropriate to me. $\endgroup$
    – Christian Hennig
    Commented May 25, 2023 at 13:28
  • $\begingroup$ Thank you for this follow up @C $\endgroup$
    – Val
    Commented Oct 12, 2023 at 13:02
  • $\begingroup$ Thank you for this follow up @ChristianHennig! This is a good thought, as these tests are constructed for independent samples. The U.S. Census Bureau creates ACS 5 yr estimates by sampling just a subset of the population. Their documentation encourages making comparisons between non-overlapping intervals (as their official estimates are averaged over these intervals). Here, I use 2017-2021 averages and 2012-2016 averages. Perhaps it is possible than some homeowners were surveyed for the ACS in both time periods. $\endgroup$
    – Val
    Commented Oct 12, 2023 at 13:09

1 Answer 1

Reset to default
0
$\begingroup$

Upon further reflection, I think it is a case where the significance may disagree.

While I had only constructed a CI for the percent change, it's also reasonable to test if that percent change differs from 0 with a zscore:

z = abs((-47.12-0)/sqrt((37.66/1.645)^2+0^2)) = 2.06... as 2.06 > 1.645, we see that this percent change is significant (as confirmed by its confidence interval not crossing 0)

It is surprising to me that the same set of numbers can produce significance or non-significance depending on the measure you choose to test (raw difference vs. percentage change). Statistics proves to be an ever-manipulatable science.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.