Best Practices with Data Wrangling before running Random Forest Predictions

Question

When doing predictions with Random Forests, we very often (or always) need to perform some pre-processing. Since I have a background of Computing and pretty much all I know from statistics comes from self-learning, this process becomes more intuition and less theory.

For instance, some of the things I get stuck with is dealing with

1. Outliers. Should we remove them all? If so, we consider an outlier based on the 3/2 rule? Should we keep them? Why?
2. When dealing with deltas of observations (as an example, suppose I'm subtracting a student grade from another), should I normalize the delta of all students or just stick to the absolute delta?
3. Sticking to the same student case, If I have cumulative data (suppose for every test I sum their last grades). Should the process be the same?
4. Do we need to apply any data transformation like log or any other? If so, when should it be done? When data range is large? What's the point of changing the domain of the data here?
5. If I have a Categorical target, can I apply regression instead of classification so the output would be (suppose the classes are 0, 1, 2) 0.132, 0.431. Would it be more accurate?
6. In what kind of problems is Random Forest more indicated? Large datasets?
7. Should I discard the less important variables? Maybe it just creates noise?

I know the pre-processing depends on the problem, data, etc. and I know there are a lot more things to look for when pre-processing. Here I'm more trying to understand the concepts behind pre-processing the data and the key points to look for when doing so. So with that in mind, what would be the key points to look for when pre-processing data? (If I didn't mention any other important point, and I'm sure a lot is missing, please consider that too). Imagine you're teaching that to your grandpa :)

Answer 1

When doing predictions with Random Forests, we very often (or always) need to perform some pre-processing.

This is not true. Random Forest is really "off-the-shelf".

Outliers. Should we remove them all? If so, we consider an outlier based on the 3/2 rule? Should we keep them? Why?

The base model used in RF is a large decision tree (usually built via CART). Decision trees are robust to outliers, because they isolate them in small regions of the feature space. Then, since the prediction for each leaf is the average (for regression) or the majority class (for classification), being isolated in separate leaves, outliers won't influence the rest of the predictions (in the case of regression for instance, they would not impact the mean of the other leaves). Bottom line: you don't care about outliers in RF. Just remove them if they are aberrant observations (e.g., due to recording errors). If they're valid cases, you can keep them.

When dealing with deltas of observations (as an example, suppose I'm subtracting a student grade from another), should I normalize the delta of all students or just stick to the absolute delta? Sticking to the same student case, If I have cumulative data (suppose for every test I sum their last grades). Should the process be the same?

The question here is not really related to RF, it is algorithm independent. The real question is what do you want to do? What are you trying to predict?

Do we need to apply any data transformation like log or any other? If so, when should it be done? When data range is large? What's the point of changing the domain of the data here?

For the same reasons you don't need to worry about outliers, you don't need to apply any kind of data transformation when using RF. For classification, you may need to apply some kind of resampling/weighing strategy if you have a class-imbalance problem, but that's it.

If I have a categorical target, can I apply regression instead of classification so the output would be (suppose the classes are 0, 1, 2) 0.132, 0.431; so would it be more accurate?

You cannot apply regression if your target is categorical.

In what kind of problems is Random Forest more indicated? Large datasets?

RF is indicated for all types of problems. People (especially in the medical field, genomics, etc.) even use it primarily for its variable importance measures. In genetics, where the guys face the "small $n$ - large $p$" problem, RF also does very well. Anyhow, Machine Learning in general requires sufficient amounts of training and testing data, though there's no general rule. If your training data represents all your concepts and if these concepts are easily capturable, a couple of hundred observations may suffice. However, if what should be learned is very complex and if some concepts are underepresented, more training data will be needed.

Should I discart the less important variables? Maybe it just creates noise?

Another nice feature of decision trees built through CART is that they automatically put aside the non-important variables (only the best splitters are selected at each split). In the seminal book by Hastie et al. (2009), the authors showed that with 100 pure noise predictors, and 6 relevant predictors, the relevant variables were still selected 50% of the time at each split. So you really don't need to worry about variable selection in RF. Of course if you know that some variables are not contributing, don't include them, but if the underlying mechanisms of the process you're studying are mostly unknown, you can include all your candidate predictors.

Answer 2

Theoretically, Random Forest is ideal as it is commonly assumed and described by Breiman and Cuttler. In practice, it is very good but far from ideal. Therefore, these questions are very valid.

1. RF are not handling outliers as ideally as it is widely assumed. They are susceptible to even a single outlier with extreme values as was shown in How are Random Forests not sensitive to outliers?, and also there are couple of papers about how heteroscedasticity affects the RF predictions. In real-life data, you may have a lot (1-2%) of such outliers caused by typos (for human-inputted data like 3200 instead of 32.00), jumps of electrical current due to induction or simply due to unexpected exposures (for IoT), heteroscedasticity, etc. These "outliers" end up in many leaves of the decision trees, pulling predictions to higher values.

2. In case of unbalanced data where large number of target_value = 0, RF tends to underestimate predictions significantly.

3. Log-transformations can improve accuracy, especially in case of very skewed data (with very long tails). See for example "Forecasting Bike Sharing Demand" by Jayant Malani et al. (pdf), and this kaggle submission.

4. RF tends assigning higher importance to variables that have larger range of values (both categorical and continuous). For example, see this blog post: Are categorical variables getting lost in your random forests?

So, data preprocessing is very important even in the case of Random Forest.

I hope, this answer frames the validity of the questions and the links will provide some answers with starting points.

Answer 3

When pre-processing data, you are generally trying to achieve the following:

A. Removal of errors from your data. If your outliers are due to data recording errors, for example, you would want to fix this in the pre-processing stage. The various rules for identifying outliers should be treated as providing initial guesses, requiring further investigation.

B. Creating variables where you have a reasonable expectation that different values of the predictor variables may by correlated with the outcome variable. This is the bit that requires domain-specific knowledge, and good variables are often constructed using ratios, differences, averages of variables, etc.

C. Modifying the data to avoid restrictive assumptions of whatever model we are fitting.

The super-cool thing about tree-based methods, like random forests, is that they require much less effort in the type C pre-processing. In particular, normalizing, removing non-error-outliers, discarding variables, and log transformations, are not generally required. But, the cost of tree-based methods is that they are data hungry, so with smaller samples (e.g., less than 10,000 case), my experience is that often a glm is going to do a better job, but this brings in the type C processing, which after 25 years of building models I still find a challenge.

Answer 4

• Outliers: In general, you want to remove outliers that are data errors (e.g. people that are allegedly 200 m tall) or correct them (e.g. maybe you are pretty sure 200 m tall means 200 cm). If the data are just garbage, it cannot help your model, but arguably, you might want to only set individual variables to missing instead of eliminating the whole record, if there are many varaibles.
• Transformations:
  • In general, monotonic 1-1 transformations of a single predictor variable are not needed, because they don't affect how RF handles the predictor.
  • When you combine multiple predictors into a new features, it may very well matter how you transform them when doing so. E.g. sales / shop visits vs. (shop visits with sales > 0) / shop visits vs. (sum(sale at visit) - average sale at visits))^2and so on will convey different information and how much they help is something that you either have to figure out by considering the logic of the situation or experimentation.
  • For outcome variables transformations really matter, but that becomes a question of what you care about (e.g. if you log-transform, you are saying that you care about relative errors instead of absolute errors).
  • Dealing with deltas of observations: This is really a question of what version of the predictor (or is it the outcome) makes sense to you / conveys information in the most logical (to you) manner. Sometimes there is something very obvious (e.g. when you have events during some observation period, transformations like events/observation time can be very obvious things to try), sometimes there isn't.
  • Cumulative data about something as a predictor: I'd normally try to not just reflect an average / summed up version, but also create a new feature indicating how long the history is (e.g. predicting future sales, you probably don't just care about the total amount a customer has spent before, nor just the average amount they've spent at a visit, but either both of those or some other equivalent features like number of previous buying events).
• Regression vs. classification for a categorical target: Yes, of course that's in principle possible. In fact, most models will in the background do that and use e.g. categorical cross-entropy loss, even if they don't tell you and primarily output a class prediction.
• Ideal setting for Random Forest: Maybe this is best described by when I would not use it:
  • When the data are not tabular (e.g. images, text, audio, video...), some form of neural network is very often a really obvious option
  • If I had to pick a single model, I probably would never pick it. When RF is good, gradient boosted decision trees are usually better, if tuned properly (except in a small data setting, but then neither is likely ideal). However, RF can be a useful contributor to an ensemble.
  • When we really understand the underlying data generating process, we probably want to specifically model it (e.g. non-linear model defined through systems of different equations to describe drug absorption, drug getting excreted, docking to a receptor, de-coupling etc.).
  • With very little data but without a clearly understood system, some simple regression model - often in a Bayesian form with informative priors based on existing knowledge - is frequently preferable.
  • When you need calibrated probabilities, RF is not that great (but not as bad as gradient boosting), although it may be possible to fix that (it's a problem with the default RF though).
  • The main argument for RF is usually that it's often kind of decent (in terms of discriminatory performance) on moderately sized to large datasets out of the box without too much hyper-parameter tuning.
• Deciding what predictors to include: In general, irrelevant or very weak predictors will (depending on the amount of data you have) hurt performance. However, removing them in a naive data driven approach can be problematic (it's a form of overfitting). Just fitting a single RF on all data, removing features with low feature importance and iteratively re-fitting is likely to overfit to the training data and not generalize so well to new unseen data. There's procedures for trying to do this more systematically (such as picking some threshold for removal and tuning this as a hyperparameter - with performance evaluated through some proper validation set-up).