2
$\begingroup$

I always had the following question: Can data having "naturally occurring hierarchical structure" be transformed to better make use of this hierarchical structure at different levels?

To illustrate this problem, I will use ZIP Codes (postal codes within the USA). Here is an example of how a ZIP Code would appear:

John Smith

ABC University

123 Fake Street

Buffalo, NY 49401

ZIP Codes contain 5 digits, e.g. "12345" - different neighborhoods in the USA are all classified under these ZIP Codes (i.e. the residents of many neighborhoods can be classified under the same ZIP Code) . If you use the full 5 digits (e.g. 12345) of a ZIP Code, it zones in on a smaller group of people. If you use the first 4 digits (e.g. 1234) of a ZIP Code, it zones in on a larger group of people - if you use the first 3 digits (e.g. 123) of a ZIP Code, it zones in on a even larger group of people, etc. In general, the ZIP Codes of people who live closer to each other are more similar than the ZIP Codes of people who live further from each other. Here are some general pictures that depict this:

enter image description here

enter image description here

Problem: Suppose you have a dataset that contains information on socio-economic status on different people, with the objective of predicting whether or not an unseen person is overweight:

  id zip_code   salary   height overweight
1  1    12345 47282.38 169.4224        yes
2  2    12346 43153.65 175.1549         no
3  3    12344 53884.52 169.1625         no
4  4    12341 36914.46 193.4863        yes
5  5    12348 48900.29 185.3250        yes
6  6    55667 63248.93 189.3762         no
7  7    55668 58288.92 167.4329        yes

Normally, the standard approach would be to use the data as is (e.g. random forest model with the R programming language):

library(randomForest)
model_5_digit = randomForest(data = my_data, overweight ~., 
                     mtry=2, ntree = 100)

However, this model is only looking at patterns amongst individuals at the 5 digit ZIP Code - perhaps there might exist more useful patterns when the ZIP code is used at the 4 digit ZIP Code (i.e. now there are fewer categories and the resulting categories are less empty - more advantageous for the model):

my_data$four_digit_zip = substr(my_data$zip_code, 1, 4)

  id zip_code   salary   height overweight four_digit_zip
1  1    12345 47282.38 169.4224        yes           1234
2  2    12346 43153.65 175.1549         no           1234
3  3    12344 53884.52 169.1625         no           1234
4  4    12341 36914.46 193.4863        yes           1234
5  5    12348 48900.29 185.3250        yes           1234
6  6    55667 63248.93 189.3762         no           5566
7  7    55668 58288.92 167.4329        yes           5566

 model_4_digit = randomForest(data = my_data[,-2], 
        overweight ~., mtry=2, ntree = 100)

But the above approach would then forfeit the ability to analyze patterns at the 5 digit level.

My Question: Suppose I were to transform the dataset so that it contains each possible permutation of the ZIP Code, and then train the same model:

my_data$one_digit_zip = substr(my_data$zip_code, 1, 1)
my_data$two_digit_zip = substr(my_data$zip_code, 1, 2)
my_data$three_digit_zip = substr(my_data$zip_code, 1, 3)
my_data$four_digit_zip = substr(my_data$zip_code, 1, 4)

  id zip_code   salary   height overweight one_digit_zip two_digit_zip three_digit_zip four_digit_zip
1  1    12345 47282.38 169.4224        yes             1            12             123           1234
2  2    12346 43153.65 175.1549         no             1            12             123           1234
3  3    12344 53884.52 169.1625         no             1            12             123           1234
4  4    12341 36914.46 193.4863        yes             1            12             123           1234
5  5    12348 48900.29 185.3250        yes             1            12             123           1234
6  6    55667 63248.93 189.3762         no             5            55             556           5566
7  7    55668 58288.92 167.4329        yes             5            55             556           5566

model_all_digit = randomForest(data = my_data, overweight ~., mtry=2, ntree = 100)

With the ability of modern statistical models(e.g. random forest: https://ui.adsabs.harvard.edu/abs/2016JPRS..114...24B/abstract) being less sensitive to multicollinearity compared to traditional statistical models (e.g. linear regression) - could this transform that I described above be suitable for data with naturally occurring hierarchical structures?

Extra: Data Visualization showing Sparsity as the Number of Digits within the ZIP Code Increase (I didn't choose enough diversity within the ZIP Codes in my example to really illustrate this phenomena , but the general idea holds):

library(ggplot2)
five = ggplot(my_data, aes(zip_code)) +
    geom_bar(fill = "#0073C2FF")  + ggtitle("5 Digits")

four = ggplot(my_data, aes(four_digit_zip)) +
    geom_bar(fill = "#0073C2FF")  + ggtitle("4 Digits")

enter image description here

Note:

  • I am not from USA, so I might have misunderstood how ZIP Codes work

  • Other examples of hierarchical variables that could be transformed this way could be "career professions" (e.g. when analyzing salaries of computer programmers : all computer programmers, the subset of computer programmers that work in the private industry, the smaller subset of computer programmers that work in the private engineers that only use java, etc.)

References:

$\endgroup$
4
  • 2
    $\begingroup$ Similar Qs are recurring on this site. Have a look at (possibly duplicate) stats.stackexchange.com/questions/146907/… and peruse other posts tagged many-categories. If you still have questions after that, come back!!! $\endgroup$ Oct 25, 2021 at 15:47
  • $\begingroup$ @ kjetil b halvorsen: thank you for your reply! I have been looking at previously posted questions on stackoverflow and crossvalidated to see if someone else might have asked a question that was similar to the one I asked. So far, I have not found anything similar to my question. At this point, I am thinking of just trying the strategy that I proposed (breaking the ZIP Code column into overlapping segments) - but I was hoping that someone might have some experience with this problem and offer some more experienced advice. thank you! $\endgroup$
    – stats_noob
    Oct 26, 2021 at 0:56
  • 2
    $\begingroup$ I don't understand how you can say there are no similar questions, I have shown you a list! Maybe you mean very specifically about zip codes? Some others: stats.stackexchange.com/questions/397969/…, stats.stackexchange.com/questions/89815/…, stats.stackexchange.com/questions/94902/… $\endgroup$ Oct 26, 2021 at 17:14
  • $\begingroup$ @ kjetil b halvorsen: thank you for your reply! I have been scanning the internet for some more official answers on this topics (multicollinearity by repeatedly creating overlapping zip code variables). I found many similar and anecdotal answers - but I am trying to find something more rigorous. Thank you so much for your time and help! I really appreciate it! $\endgroup$
    – stats_noob
    Oct 26, 2021 at 17:19

2 Answers 2

1
$\begingroup$

There are many similar questions, although you do not seem to realize that, maybe because you insist on using random forest as a model. Maybe you should try some more standard regression model first? Then you can always, later, see if you can do better with a random forest, when you have a baseline model to compare with.

At Principled way of collapsing categorical variables with many levels?, fused lasso is proposed as a solution. This is implicitly grouping together zip codes that are similar in its effect on the response variable, so you are making a hierarchy from the data, not superimposing a known hierarchy. I would start with something like that.

Then, if you want to continue with random forests, you need to find an implementation that works well with categorical data with very many levels. See my answer at Random Forest Regression with sparse data in Python, where there is references to implementations which is said to work well in such a setting.

$\endgroup$
1
  • 1
    $\begingroup$ I would go with the per-zip-code median family income as a single continuous variable. This is provided by the U.S. Census though sometimes has to be purchased. $\endgroup$ Oct 28, 2021 at 13:16
0
$\begingroup$

I recommend having a look at Poincare embeddings ( https://github.com/facebookresearch/poincare-embeddings) which are designed for hierarchical graph structures .

In the case of ZIP codes there is if course a natural ( non-trained) embedding in 2D ; latitude and longitude. I recommend using that as 2 features and get rid of zip codes category. That also solves sparsity issue on rare codes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.