Data with Hierarchical Structure and Multicollinearity (E.g. ZIP Postal Codes)

Question

I always had the following question: Can data having "naturally occurring hierarchical structure" be transformed to better make use of this hierarchical structure at different levels?

To illustrate this problem, I will use ZIP Codes (postal codes within the USA). Here is an example of how a ZIP Code would appear:

John Smith

ABC University

123 Fake Street

Buffalo, NY 49401

ZIP Codes contain 5 digits, e.g. "12345" - different neighborhoods in the USA are all classified under these ZIP Codes (i.e. the residents of many neighborhoods can be classified under the same ZIP Code) . If you use the full 5 digits (e.g. 12345) of a ZIP Code, it zones in on a smaller group of people. If you use the first 4 digits (e.g. 1234) of a ZIP Code, it zones in on a larger group of people - if you use the first 3 digits (e.g. 123) of a ZIP Code, it zones in on a even larger group of people, etc. In general, the ZIP Codes of people who live closer to each other are more similar than the ZIP Codes of people who live further from each other. Here are some general pictures that depict this:

Problem: Suppose you have a dataset that contains information on socio-economic status on different people, with the objective of predicting whether or not an unseen person is overweight:

  id zip_code   salary   height overweight
1  1    12345 47282.38 169.4224        yes
2  2    12346 43153.65 175.1549         no
3  3    12344 53884.52 169.1625         no
4  4    12341 36914.46 193.4863        yes
5  5    12348 48900.29 185.3250        yes
6  6    55667 63248.93 189.3762         no
7  7    55668 58288.92 167.4329        yes

Normally, the standard approach would be to use the data as is (e.g. random forest model with the R programming language):

library(randomForest)
model_5_digit = randomForest(data = my_data, overweight ~., 
                     mtry=2, ntree = 100)

However, this model is only looking at patterns amongst individuals at the 5 digit ZIP Code - perhaps there might exist more useful patterns when the ZIP code is used at the 4 digit ZIP Code (i.e. now there are fewer categories and the resulting categories are less empty - more advantageous for the model):

my_data$four_digit_zip = substr(my_data$zip_code, 1, 4)

  id zip_code   salary   height overweight four_digit_zip
1  1    12345 47282.38 169.4224        yes           1234
2  2    12346 43153.65 175.1549         no           1234
3  3    12344 53884.52 169.1625         no           1234
4  4    12341 36914.46 193.4863        yes           1234
5  5    12348 48900.29 185.3250        yes           1234
6  6    55667 63248.93 189.3762         no           5566
7  7    55668 58288.92 167.4329        yes           5566

 model_4_digit = randomForest(data = my_data[,-2], 
        overweight ~., mtry=2, ntree = 100)

But the above approach would then forfeit the ability to analyze patterns at the 5 digit level.

My Question: Suppose I were to transform the dataset so that it contains each possible permutation of the ZIP Code, and then train the same model:

my_data$one_digit_zip = substr(my_data$zip_code, 1, 1)
my_data$two_digit_zip = substr(my_data$zip_code, 1, 2)
my_data$three_digit_zip = substr(my_data$zip_code, 1, 3)
my_data$four_digit_zip = substr(my_data$zip_code, 1, 4)

  id zip_code   salary   height overweight one_digit_zip two_digit_zip three_digit_zip four_digit_zip
1  1    12345 47282.38 169.4224        yes             1            12             123           1234
2  2    12346 43153.65 175.1549         no             1            12             123           1234
3  3    12344 53884.52 169.1625         no             1            12             123           1234
4  4    12341 36914.46 193.4863        yes             1            12             123           1234
5  5    12348 48900.29 185.3250        yes             1            12             123           1234
6  6    55667 63248.93 189.3762         no             5            55             556           5566
7  7    55668 58288.92 167.4329        yes             5            55             556           5566

model_all_digit = randomForest(data = my_data, overweight ~., mtry=2, ntree = 100)

With the ability of modern statistical models(e.g. random forest: https://ui.adsabs.harvard.edu/abs/2016JPRS..114...24B/abstract) being less sensitive to multicollinearity compared to traditional statistical models (e.g. linear regression) - could this transform that I described above be suitable for data with naturally occurring hierarchical structures?

Extra: Data Visualization showing Sparsity as the Number of Digits within the ZIP Code Increase (I didn't choose enough diversity within the ZIP Codes in my example to really illustrate this phenomena , but the general idea holds):

library(ggplot2)
five = ggplot(my_data, aes(zip_code)) +
    geom_bar(fill = "#0073C2FF")  + ggtitle("5 Digits")

four = ggplot(my_data, aes(four_digit_zip)) +
    geom_bar(fill = "#0073C2FF")  + ggtitle("4 Digits")

Note:

• I am not from USA, so I might have misunderstood how ZIP Codes work

• Other examples of hierarchical variables that could be transformed this way could be "career professions" (e.g. when analyzing salaries of computer programmers : all computer programmers, the subset of computer programmers that work in the private industry, the smaller subset of computer programmers that work in the private engineers that only use java, etc.)

References:

• https://studylib.net/doc/8397761/zip-code-page
• https://en.wikipedia.org/wiki/List_of_ZIP_Code_prefixes
• https://ny.curbed.com/2019/11/18/20970450/nyc-top-10-expensive-zip-codes
• https://en.wikipedia.org/wiki/ZIP_Code

Answer 1

There are many similar questions, although you do not seem to realize that, maybe because you insist on using random forest as a model. Maybe you should try some more standard regression model first? Then you can always, later, see if you can do better with a random forest, when you have a baseline model to compare with.

At Principled way of collapsing categorical variables with many levels?, fused lasso is proposed as a solution. This is implicitly grouping together zip codes that are similar in its effect on the response variable, so you are making a hierarchy from the data, not superimposing a known hierarchy. I would start with something like that.

Then, if you want to continue with random forests, you need to find an implementation that works well with categorical data with very many levels. See my answer at Random Forest Regression with sparse data in Python, where there is references to implementations which is said to work well in such a setting.

Answer 2

I recommend having a look at Poincare embeddings ( https://github.com/facebookresearch/poincare-embeddings) which are designed for hierarchical graph structures .

In the case of ZIP codes there is if course a natural ( non-trained) embedding in 2D ; latitude and longitude. I recommend using that as 2 features and get rid of zip codes category. That also solves sparsity issue on rare codes.