2
$\begingroup$

I have a data set where my observations are geographically referenced by longitude & latitude, as well as by township & range. If you're unfamiliar with township & range:

The land is divided into survey townships of roughly 36 square miles. This is done by establishment of township and range lines. Township lines run parallel to the baseline, while range lines are true meridians. (https://en.wikipedia.org/wiki/Public_Land_Survey_System)

I want to run two different configurations of a Random Forest model. One where I account for spatial autocorrelation by including longitude & latitude as predictors, and one where I use township & range for the same. The point is to develop two models of different "spatial resolution", where the one with longitude & latitude is the high-precision case and the one with range & township is the lower-precision case (this would in a way resemble a county-level fixed effect).

At the moment, I have both these variables encoded as numeric variables (I am using R). For longitude & latitude, the precision can be infinitely increased by increasing the number of decimals, so this seems reasonable. However, range & township may only be integers (there's no such thing as township = 5.5). I have tried to encode them as integers, but this doesn't seem to make any difference, as R seems to treat them in the same manner as normal numeric variables. If I encode them as factors, the variable importance gets way more messy since each category acts as a dummy variable, and I am not sure whether it's "more correct" anyway.

My question is: Would there be any problems related to keeping the township & range variables as numeric in this case? If so, does anyone have any suggestions on how to store these variables in a different way?

PS, I have read this post: GLMM: Elevation as numerical or factor in model?, but since random forests are capable of modelling non-linear relationships, the accepted answer does not seem to be exactly what I'm after.

$\endgroup$

1 Answer 1

1
$\begingroup$

I think it will be reasonable to consider encoding township and ranges in a way other than simply treating them as integers. While it is safe to assume that RF are able to encapsulate non-linear relations, it will still be rather difficult to interpreter results from a integer encoding. In addition, spurious effects might be very prevalent. For example, if a relatively low-counts categorical level has an assigned numerical value that is close to the assigned numerical variable of high-counts categorical level, it will be probably pooled together with it even if they encode relatively different information; and that would be just due to the encoding, another (random) integer encoding might give something else.

Similarly, as you have correctly deduced, because township and range have high cardinality (i.e. they have many distinct variables) as well as they have an inherit hierarchy, using them directly as factors quickly becomes unwieldy. The main problems are: 1. We might overfit out data when concerned about the influence of very unusual/rare factors and 2. with thousands of factor levels is difficult to estimate the overall importance of a factor as a whole. Bonus problem: (3.) We might not respect the hierarchy among two level (e.g. in the UK, where the post-codes (e.g. EC1Y 8LX) have an outward code (EC1Y) and inward code (8LX) part, it makes little sense to analyse inward post codes in absence of outward post codes).

I would suggest looking into target encoding (Micci-Barreca (2001) A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems) (or other target aware encodings like the James-Stein Encoder or the M-Estimator encoding). The basic idea is that a discrete factor variable is replace by a (regularised) average of the response variable. Here is a quick example: if we model a response variable y and we have three options of a variable Sex: Female/Male/Other, each of them represent 53%, 46% and 1% of our sample respectively. We will then create a new "numeric version" of the variable Sex where Female will be replaced with the mean of y for Female correspondents, $\mu_{y;Female}$, Male will be replaced with the mean of y for Male correspondents $\mu_{y;Male}$ (or something a bit closer to the overall average of y, $\mu_y$ due to regularisation) and Other will be replaced with the mean of y for Other correspondents, $\mu_{y;Other}$ (or likely something even closer to $\mu_y$, due to even stronger regularisation as the proportion is smaller).

That and a bunch of other goodies can be found in the vtreat package that has a number of pretreatment step for data. They have a number of vignettes that touch upon a number of different variable pre-processing issues.

Side-note: There are other encoding schema like Binary Encoding (i.e. turn everything into integers and then make $p$ distinct variable where they hold the $p$ digits 0/1 required to encode the integer) and Feature Hashing (Weinberger et al. 2009 Feature Hashing for Large Scale Multitask Learning). I do not touch upon them here as variable importance and influence is even harder to interpreter in these use cases. They are mostly dimensionality reduction pre-processing steps.

$\endgroup$
5
  • 1
    $\begingroup$ Good references. I agree that it makes much more sense to treat numeric information that are, in reality, categorical as categorical. Impact coding is another approach which may or may not be redundant with your suggestions. It's certainly similar in spirit. win-vector.com/blog/tag/impact-coding $\endgroup$
    – user234562
    Apr 13, 2020 at 17:59
  • $\begingroup$ @user332577: You have the correct idea. Impact encoding is very similar to target encoding. The main difference is that with impact encoding we subtract the "overall sample mean, $\mu_y$, while with target encoding the mean is still there. As the mean difference between impact and target encoding is a level shift, they encode the same information. I have seen the terms "impact" and "target" encoding, used interchangeably. I think it is mostly an author preference. (I hope this clarifies your comment). $\endgroup$
    – usεr11852
    Apr 13, 2020 at 21:51
  • $\begingroup$ @usεr11852 Thanks for valuable input. I will have a look at the references you provided. Just one question right away: When you refer to regularisation in your example, do you mean that, for instance, the mean for female is multiplied by 0.53? $\endgroup$
    – veghokstvd
    Apr 14, 2020 at 11:23
  • 1
    $\begingroup$ Think of it this way: if we have a large number of samples in a subcategory $A$ probably $\mu_{y,A}$ is a good representation of what is really happening with $A$'s samples and indeed $A$ members' values for $y$ should close to $\mu_{y,A}$. On the other hand, if for a subcategory $B$ we have only a handful of members we are less willing to believe that the average $\mu_{y,B}$ is a good representation of what is really happening with members of $B$; we will therefore regularise the value of $\mu_{y,B}$ such that it is closer to the overall mean $\mu_y$. $\endgroup$
    – usεr11852
    Apr 14, 2020 at 12:06
  • 1
    $\begingroup$ We will therefore not "multiply but 0.53" but rather "pull" $\mu_{y;Female}$ towards the overall sample mean $\mu_{y}$ with less force than we would "pull" $\mu_{y;Other}$. In the reference provided: Micci-Barreca (2001) this can be first seen in Equation (3), here the force of the "pull" towards the sample mean $\mu_y$ is captured in $\lambda$. James-Stein estimators have essentially the same form too. $\endgroup$
    – usεr11852
    Apr 14, 2020 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.