7
$\begingroup$

In seeing this New York Times graphic on Where Same Sex Couples Live, it appears that the low population counties have the most variation (comparing North Dakota and Ohio, for instance). Presumably some of that variation is because of the lower sample sizes. What is the proper way to adjust for that, especially given that this is from sampled Census data?

enter image description here

I tried calculating a $z$ score of the mean as in Ratio that accounts for different sample sizes. The resulting scores seem exaggerated (-20 to 200), and I'm wondering if it's because I was using the number of households as the sample size instead of the number of sampled households. That is, the census only samples about 1% of households (based on a report of ~3 million ACS surveys), so perhaps the baseline sample size should be 1/100 of the number of households in the county. The $z$ scores are then reduced by a factor of 10, and the values are shown here (still truncating the high end of the range).

enter image description here

The distribution of the proportions is slightly skewed, and I haven't adjusted it. Presumably some of the skew are real outliers and not systematic variation.

enter image description here

The NYT data lives in a TSV file though some of the county names are missing (use the FIPS codes instead). Also, their data is adjusted to account for miscoded surveys.

I'm essentially trying to use scoring comparable to a funnel plot, and here's what my funnel plot looks like with the adjusted sample sizes.

enter image description here

Main question: What to use as the sample size for this data in computing the $z$ score? Underlying question: Is this the right way to standardize the proportions for visual comparison?

$\endgroup$
2
$\begingroup$

After talking to local statisticians and not seeing any other answers, I can provide some answer. I'm also happy to remove the question if commentators think it is too narrow.

The number of respondents is the right sample size for the score computations. I was using 1%, and I've since learned that 2/3 of 1% is a better estimate of the response rate. I can get state level sample sizes from the Census Bureau. I've also verified the data comes from the American Community Survey rather than the general census, which doesn't ask relationship questions.

It was also suggested to exclude the far outliers when computing the grand mean with the idea that those locations are categorically different from the general population of counties.

Another technique for handling variation due to small samples is Small Area Estimation, which can be thought of as a kind of weighted smoother.

Though I had forgotten the source, I now realize my inspiration for this line of exploration was Howard Wainer's discussion of similar issues with cancer rates by county and test results by school, collected in Picturing the Uncertain World.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.