I am facing a problem with my PhD Thesis in Archaeology. After analysing the percentages of an animal (variable: Observation) by grouping them by each variable in the following columns (e.g., Median % of Cattle in Urban sites), I am interested in knowing which of these features is more important, rather than simply exploring different distributions. The problem is that all the variables (except the observation) are categorical, so I do not know precisely how to proceed. The question I am interested in answering is: 'Which is the most important factor in choosing this animal? Region, Geography, site type?'

I started reading a bit about decision trees, random forests, etc. but I am unsure whether I can use these types of analyses or how should I reshape my data in order to use them.

Any advice would be appreciated, I know this is probably a vague question, but unfortunately I am not a statistician and I am a bit stuck with this issue.

Thank you in advance!

Observation Region Geography_Type Site_Type
40 North Coast Urban
25 South Mountain Farm
... ... ... ...

1 Answer 1


All feature importance measures come down to assessing how much the classification degrades if a particular predictor is either completely removed from the model, or randomly permuted. If a model that removes (or randomly permutes) a particular predictor yields predictions that are as good as a model that retains that predictor, then the predictor can't be that important. The difference in predictive performance is then typically normalized in some way.

Thus, a measure of feature importance really comes down to choosing an evaluation metric (if you predict numerical observations, this could be the Mean Squared Error, or in other cases, it could be Gini impurity or similar), deciding whether to assess this in-sample, out-of-bag or with a holdout sample, and then running multiple models, removing or permuting predictors.

Some of these approaches are already provided in standard tools, e.g., in Random Forests (as in the eponymous R package), where this is a very simple and natural extension of the core functionality. In other cases (e.g., OLS), you can do the same thing, but you may need to code it yourself.

In any case, there is really nothing particular about categorical predictors here, beyond having a model that can deal with them.

  • $\begingroup$ Thank you for your answer. I talked about categorical predictors as all the examples I have read on Random Forests were using numerical variables. Would use one hot encoding help? $\endgroup$
    – Roberto
    Nov 30, 2022 at 17:02
  • $\begingroup$ Random Forests can certainly use categorical predictors, and this is quite standard. Typically, the interface will take care of the encoding and do one-hot automatically (or something else). $\endgroup$ Dec 3, 2022 at 8:39
  • $\begingroup$ precisely why I am sticking to R for this. In Python I would have to do the one-hot manually. The problem is (since I am new to this, I admit it is my fault) that I can't understand what should be the parameters provided to the function. What I am interested in is farming practices, so I would like to know more about the logic (the categorical variables in my data frame) behind choosing certain animals (my numerical variable) in certain places. $\endgroup$
    – Roberto
    Dec 5, 2022 at 8:35
  • $\begingroup$ You could ask at StackOverflow in the R tag about how to feed your favorite model fitting function. Providing a Minimal Working Example will help you get better answers. Feel free to add a comment here pointing to your question; if I find the time, I can try having a look. $\endgroup$ Dec 5, 2022 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.