# Data with Hierarchical Structure and Multicollinearity (E.g. ZIP Postal Codes)

I always had the following question: Can data having "naturally occurring hierarchical structure" be transformed to better make use of this hierarchical structure at different levels?

To illustrate this problem, I will use ZIP Codes (postal codes within the USA). Here is an example of how a ZIP Code would appear:

John Smith

ABC University

123 Fake Street

Buffalo, NY 49401

ZIP Codes contain 5 digits, e.g. "12345" - different neighborhoods in the USA are all classified under these ZIP Codes (i.e. the residents of many neighborhoods can be classified under the same ZIP Code) . If you use the full 5 digits (e.g. 12345) of a ZIP Code, it zones in on a smaller group of people. If you use the first 4 digits (e.g. 1234) of a ZIP Code, it zones in on a larger group of people - if you use the first 3 digits (e.g. 123) of a ZIP Code, it zones in on a even larger group of people, etc. In general, the ZIP Codes of people who live closer to each other are more similar than the ZIP Codes of people who live further from each other. Here are some general pictures that depict this:

Problem: Suppose you have a dataset that contains information on socio-economic status on different people, with the objective of predicting whether or not an unseen person is overweight:

  id zip_code   salary   height overweight
1  1    12345 47282.38 169.4224        yes
2  2    12346 43153.65 175.1549         no
3  3    12344 53884.52 169.1625         no
4  4    12341 36914.46 193.4863        yes
5  5    12348 48900.29 185.3250        yes
6  6    55667 63248.93 189.3762         no
7  7    55668 58288.92 167.4329        yes


Normally, the standard approach would be to use the data as is (e.g. random forest model with the R programming language):

library(randomForest)
model_5_digit = randomForest(data = my_data, overweight ~.,
mtry=2, ntree = 100)


However, this model is only looking at patterns amongst individuals at the 5 digit ZIP Code - perhaps there might exist more useful patterns when the ZIP code is used at the 4 digit ZIP Code (i.e. now there are fewer categories and the resulting categories are less empty - more advantageous for the model):

my_data$$four_digit_zip = substr(my_data$$zip_code, 1, 4)

id zip_code   salary   height overweight four_digit_zip
1  1    12345 47282.38 169.4224        yes           1234
2  2    12346 43153.65 175.1549         no           1234
3  3    12344 53884.52 169.1625         no           1234
4  4    12341 36914.46 193.4863        yes           1234
5  5    12348 48900.29 185.3250        yes           1234
6  6    55667 63248.93 189.3762         no           5566
7  7    55668 58288.92 167.4329        yes           5566

model_4_digit = randomForest(data = my_data[,-2],
overweight ~., mtry=2, ntree = 100)


But the above approach would then forfeit the ability to analyze patterns at the 5 digit level.

My Question: Suppose I were to transform the dataset so that it contains each possible permutation of the ZIP Code, and then train the same model:

my_data$$one_digit_zip = substr(my_data$$zip_code, 1, 1)
my_data$$two_digit_zip = substr(my_data$$zip_code, 1, 2)
my_data$$three_digit_zip = substr(my_data$$zip_code, 1, 3)
my_data$$four_digit_zip = substr(my_data$$zip_code, 1, 4)

id zip_code   salary   height overweight one_digit_zip two_digit_zip three_digit_zip four_digit_zip
1  1    12345 47282.38 169.4224        yes             1            12             123           1234
2  2    12346 43153.65 175.1549         no             1            12             123           1234
3  3    12344 53884.52 169.1625         no             1            12             123           1234
4  4    12341 36914.46 193.4863        yes             1            12             123           1234
5  5    12348 48900.29 185.3250        yes             1            12             123           1234
6  6    55667 63248.93 189.3762         no             5            55             556           5566
7  7    55668 58288.92 167.4329        yes             5            55             556           5566

model_all_digit = randomForest(data = my_data, overweight ~., mtry=2, ntree = 100)


With the ability of modern statistical models(e.g. random forest: https://ui.adsabs.harvard.edu/abs/2016JPRS..114...24B/abstract) being less sensitive to multicollinearity compared to traditional statistical models (e.g. linear regression) - could this transform that I described above be suitable for data with naturally occurring hierarchical structures?

Extra: Data Visualization showing Sparsity as the Number of Digits within the ZIP Code Increase (I didn't choose enough diversity within the ZIP Codes in my example to really illustrate this phenomena , but the general idea holds):

library(ggplot2)
five = ggplot(my_data, aes(zip_code)) +
geom_bar(fill = "#0073C2FF")  + ggtitle("5 Digits")

four = ggplot(my_data, aes(four_digit_zip)) +
geom_bar(fill = "#0073C2FF")  + ggtitle("4 Digits")


Note:

• I am not from USA, so I might have misunderstood how ZIP Codes work

• Other examples of hierarchical variables that could be transformed this way could be "career professions" (e.g. when analyzing salaries of computer programmers : all computer programmers, the subset of computer programmers that work in the private industry, the smaller subset of computer programmers that work in the private engineers that only use java, etc.)

References:

• Similar Qs are recurring on this site. Have a look at (possibly duplicate) stats.stackexchange.com/questions/146907/… and peruse other posts tagged many-categories. If you still have questions after that, come back!!! Oct 25, 2021 at 15:47
• @ kjetil b halvorsen: thank you for your reply! I have been looking at previously posted questions on stackoverflow and crossvalidated to see if someone else might have asked a question that was similar to the one I asked. So far, I have not found anything similar to my question. At this point, I am thinking of just trying the strategy that I proposed (breaking the ZIP Code column into overlapping segments) - but I was hoping that someone might have some experience with this problem and offer some more experienced advice. thank you! Oct 26, 2021 at 0:56
• I don't understand how you can say there are no similar questions, I have shown you a list! Maybe you mean very specifically about zip codes? Some others: stats.stackexchange.com/questions/397969/…, stats.stackexchange.com/questions/89815/…, stats.stackexchange.com/questions/94902/… Oct 26, 2021 at 17:14
• @ kjetil b halvorsen: thank you for your reply! I have been scanning the internet for some more official answers on this topics (multicollinearity by repeatedly creating overlapping zip code variables). I found many similar and anecdotal answers - but I am trying to find something more rigorous. Thank you so much for your time and help! I really appreciate it! Oct 26, 2021 at 17:19