# Why can't we reliably compare states?

I came across this catchy visualization on reddit showing the percentage of adults reporting binge or heavy drinking in the US: https://www.reddit.com/r/dataisbeautiful/comments/nve04b/oc_percentage_of_adults_reporting_binge_or_heavy/

Several users including the author warn against comparing states: "Caution should be used when comparing these estimates across state lines. The model used to create these estimates includes a state-level factor that limits comparability between neighboring counties of adjacent states". Indeed, one can see the difference between Texas and Oklahoma being at least strange.

I don't fully understand how the inclusion of a factor such as state in addition to county in a presumably additive regression model may lead to misleading interpretations. Is it a sample size problem whereby some counties have very low data? Or is it the interpretation of the coefficients?

• Do you know anything about the model used? The uploader's original comment only says that "Python was used for data processing". Jul 5, 2021 at 15:33
• Unfortunately no. I'm just wondering what are some of the things that could make the difference btween states more different than one would expect beyond different state-level regulations (again, see TX vs OK) Jul 5, 2021 at 15:56
• This is one of many examples where the statistical methods are kept a deep secret for no good reason. Did they report the source of the original data, at least? Jul 5, 2021 at 18:38
• @Solarlump I can't answer your question. But the posters of this visualization appear not to have done the modeling that resulted in the estimates of heavy drinking. I think that they simply plotted data reported here countyhealthrankings.org/explore-health-rankings Jul 8, 2021 at 18:01
• @Solarlump Here is link to reference to model(s) used to make estimates of prevalence using data from Behavioral Risk Factor Surveillance. academic.oup.com/aje/article/179/8/1025/109078 Jul 8, 2021 at 18:20

Those who posted this visualization appear not to have done the modeling that resulted in the estimates of heavy drinking displayed in the visualization. They have simply plotted data reported here:

https://www.countyhealthrankings.org/explore-health-rankings/measures-data-sources/county-health-rankings-model/health-factors/health-behaviors/alcohol-drug-use/excessive-drinking

The source of the data is the Behavioral Risk Factor Surveillance System (also known as BRFSS). This project is managed by the Centers for Disease Control and Prevention. As described at the linked website:

“The Behavioral Risk Factor Surveillance System (BRFSS) is a state-based random digit dial (RDD) telephone survey that is conducted annually in all states, the District of Columbia, and U.S. territories. Data obtained from the BRFSS are representative of each state’s total non-institutionalized population over 18 years of age and have included more than 400,000 annual respondents with landline telephones or cellphones since 2011.”

In specific, the data for the measure “excessive drinking” used data from the 2018 Behavioral Risk Factor Surveillance System survey.

Although the BRFSS survey is large, there are counties that have only a small number of participants. The estimate of the percentage of people who drink “excessively” by county is based on a multilevel model. Thus, as described at the website:

” Statistical modeling is used to obtain more informed and reliable estimates than survey data alone can provide. Our Excessive Drinking estimates are produced from one year of survey data and are created using complex statistical modeling. Modeling generates more stable estimates for places with small numbers of residents or survey responses.”

A reference to a published paper that provides technical detail on the modeling approach for making the “small area estimates” is as follows:

Zhang X, Holt JB, Lu H, Wheaton AG, Ford ES, Greenlund K, Croft JB. Multilevel regression and poststratification for small-area estimation of population health outcomes: a case study of chronic obstructive pulmonary disease prevalence using the Behavioral Risk Factor Surveillance System. American Journal of Epidemiology. 2014; 179(8):1025–1033.

The full text is available at: