Validity of combining surveys and sociodemographics to estimate miles walked in census tracts

Question

I ran across this paper that describes an interesting approach to combining survey and sociodemographic data:

Salon. Estimating Total Miles Walked and Biked by Census Tract in California. State of California Department of Transportation.

As stated, their main contribution seems to be "a new method of estimating pedestrian and cyclist activity levels at a fine geographic scale."

A quote from the document best explains their method:

(1) First, we use cluster analysis to assign census tracts to neighborhood types based on built environment characteristics, and (2) we calculate miles biked and miles walked for each travel survey respondent. All survey respondents are included, and those who do not report cycling or walking are assumed to walk and bike zero miles. (3) We then assign each survey respondent to a category based on their age, gender, and home neighborhood type. (4) Finally, we calculate average miles biked and miles walked for each category, and use census data to expand these average distances walked and biked to represent population totals.

Step labels in parentheses were added by myself.

They used this equation to calculate total miles for each tract:

TotalMiles$_{tract}=\Sigma_{i=1}^{10} $SurveyAvgMiles$_i\times $2010CensusTractPopulation$_i$

where $i$ indexes gender-age group categories and each tract is classified as a neighborhood type.

The authors alude to several other studies in their domain that used a similar approach.

However, the only justification that I could find for the validity of their methodology was that the results obtained using two different surveys, after adjusting for "differences in survey response" were "broadly consistent."

It seems to me that they are still lacking a source of truth against which their census tract estimates can be evaluated.

How do we know that this new methodology is sound?

Answer 1

This seems like a version of small area estimation problem where they used indirect domain estimation. This is an OK technique (and easy to implement -- you just keep taking ratios of one thing over another and multiply it by the third thing), but it does not allow assessing the quality of the model. Better small area models are based on regression techniques, where you can analyze residuals to say how well your model $$ \mbox{miles walked or biked} = \mbox{cellwise constant}( \mbox{age group} \otimes \mbox{gender}) $$ fits the data. You can always ask something like "Why isn't there the area under parks or the number of bike racks in the model? Or household income?" Likewise, the quality of cluster analysis of census tracts into just four clusters is not assessed -- I imagine you can find more structure in 8000 points. So this analysis could be tightened a bit or two. The greatest strength of the proper SAE models is that they allow assessment of uncertainty (standard errors), while this approach didn't. But this is as far as most GIS-type analysis go, anyway.

Answer 2

I'm the author of that report! Thanks for checking it out. The method is indeed extremely simple - no statistics required at all. I use regression and other econometric techniques in most of my other work, but for this research question (how much biking and walking is going on in each census tract?), I couldn't see a great way to do this. That said, you are correct that there is no way to ground-truth the results, and our estimates are rough.

A basic regression that aims to explain variation in miles walked at the individual level, for instance, has an R-squared of only 0.04 when I include only gender-age-neighborhood categories as the independent variables. This isn't a good indicator of how accurate the averages in each category are, however, and the accuracy of the averages is what matters here. Goodness of fit statistics will always be much lower when trying to predict individual, idiosyncratic behavior choices than they will be when predicting averages.

Nobody really knows how much biking or walking happens in each tract, and our point with this paper is that this really simple method offers a better guess than we've seen elsewhere. The simplicity is attractive because it makes the work easily replicable, even by bike/ped planners who are not experts in statistics. That said, suggestions for improvement are very welcome!

A quick additional response on the cluster analysis - we did not go into this in the paper, but the cluster results become quite unstable when we identify more than four clusters. In addition to the instability of the clusters, tracts that we know (from personal experience) are extremely similar to one another begin to get classified in different groups. This is why we use the four cluster solution for the neighborhood types in the analysis.