22
$\begingroup$

I am building a model and I think that geographic location is likely to be very good at predicting my target variable. I have the zip code of each of my users. I am not entirely sure about the best way to include zip code as a predictor feature in my model though. Although zip code is a number, it doesn't mean anything if the number goes up or down. I could binarize all 30,000 zip codes and then include them as features or new columns (e.g., {user_1: {61822: 1, 62118: 0, 62444: 0, etc.}}. However, this seems like it would add a ton of features to my model.

Any thoughts on the best way to handle this situation?

$\endgroup$
  • 1
    $\begingroup$ Just a thought.. but, if zipcodes are distributed geographically then you could geographically represent zipcodes in a map and represent them with their location. With that you could also see which zipcodes are closeser.. $\endgroup$ – Manuel Apr 23 '14 at 18:40
  • 1
    $\begingroup$ See stats.stackexchange.com/questions/146907/… $\endgroup$ – kjetil b halvorsen Mar 17 at 10:14
13
$\begingroup$

One of my favorite uses of zip code data is to look up demographic variables based on zipcode that may not be available at the individual level otherwise...

For instance, with http://www.city-data.com/ you can look up income distribution, age ranges, etc., which might tell you something about your data. These continuous variables are often far more useful than just going based on binarized zip codes, at least for relatively finite amounts of data.

Also, zip codes are hierarchical... if you take the first two or three digits, and binarize based on those, you have some amount of regional information, which gets you more data than individual zips.

As Zach said, used latitude and longitude can also be useful, especially in a tree based model. For a regularized linear model, you can use quadtrees, splitting up the United States into four geographic groups, binarized those, then each of those areas into four groups, and including those as additional binary variables... so for n total leaf regions you end up with [(4n - 1)/3 - 1] total variables (n for the smallest regions, n/4 for the next level up, etc). Of course this is multicollinear, which is why regularization is needed to do this.

$\endgroup$
16
$\begingroup$

There's 2 good options that I've seen:

  1. Convert each zipcode to a dummy variable. If you have a lot of data, this can be a quick and easy solution, but you won't be able to make predictions for new zip codes. If you're worried about the number of features, you can add some regularization to your model to drop some of the zipcodes out of the model.
  2. Use the latitude and longitude of the center point of the zipcode as variables. This works really well in tree-based models, as they can cut up the latitude/longitude grid into regions that are relevant to your target variable. This will also allow you to make predictions for new zipcodes, and doesn't require as much data to get right. However, this won't work well for linear models.

Personally, I really like tree-based models (such as random forest or GBMs), so I almost always choose option 2. If you want to get really fancy, you can use the lat/lon of the center of population for the zipcode, rather than the zipcode centroid. But that can be hard to get ahold of.

$\endgroup$
  • $\begingroup$ Definitely will go for the 2 proposal. $\endgroup$ – andilabs Apr 24 '14 at 8:51
  • $\begingroup$ #2 also appears to works with a GAM $\endgroup$ – Affine Apr 25 '14 at 17:59
4
$\begingroup$

I dealt with something similar when training a classifier that used native language as a feature (how do you measure similarity between English and Spanish?) There are lots of methods out there for determining similarity among non-categorical data.

It depends on your data, but if you find that geographic distance from a zip code is not as important as whether a given input contains particular zip codes, then non-categorical methods might help.

$\endgroup$
4
$\begingroup$

If you are calculating distance between records, as in clustering or K-NN, distances between zipcodes in their raw form might be informative. 02138 is much closer to 02139, geographically, than it is to 45809.

$\endgroup$
  • $\begingroup$ also for tree models like random forest - which in some respects are similar to K-NN $\endgroup$ – captain_ahab Feb 24 '17 at 2:06
3
$\begingroup$

You could transform your zip code into a nominal variable (string/factor). However, as far as I remember, zip code might contain other information like county, region, etc. What I would do is to understand how zip code encodes information and decode that into multiple features.

Anyway letting zip code as a numeric variable is not a good idea since some models might consider the numeric ordering or distances as something to learn.

$\endgroup$
  • $\begingroup$ Thanks for the answer! However, even if zip code is a string or factor, aren't I essentially just dummy coding zip code (i.e., creating 30,000 binarized features)? I know R does this under the hood but it has to be explicitly done in scikit learn. $\endgroup$ – captain_ahab Apr 23 '14 at 18:39
3
$\begingroup$

I would make a choropleth map of your model's residuals at the zip code level.

The result is called a spatial residual map and it may help you choose a new explanatory variable to include in your model. This approach is called exploratory spatial data analysis (ESDA) .

One potential workflow:

  1. for each zip code get the average residual
  2. make a choropleth map to see the geographic distribution of the residuals
  3. look for patterns that might be explained by a new explanatory variable. For example, if you see all suburban or southern or beach zipcodes with high residuals then you can add a regional dummy variable defined by the relevant zipcode grouping, or if you see high residuals for high income zipcodes then you can add an income variable.
$\endgroup$
-2
$\begingroup$

You can featurize the Zipcodes using the above techniques, but let me suggest an alternative. Suppose we have binary class labels. And in data we have "n" zip codes. Now we take the probability of occurence of each pincode in data, provided some class label(either 1 or zero). So, lets say for a zipcode "j" ------>>>> We get a probability P_j as: no. of occurences of "j" / Total no of occurences of "j", when class label is 1 or 0. This way we can convert it into a very nice proabilistic interpretation.

$\endgroup$
  • 5
    $\begingroup$ This answer is not very clear. $\endgroup$ – Michael Chernick Feb 11 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.