Best Practices with Data Wrangling before running Random Forest Predictions

When doing predictions with Random Forests, we very often (or always) need to perform some pre-processing. Since I have a background of Computing and pretty much all I know from statistics comes from self-learning, this process becomes more intuition and less theory.

For instance, some of the things I get stuck with is dealing with

1. Outliers. Should we remove them all? If so, we consider an outlier based on the 3/2 rule? Should we keep them? Why?
2. When dealing with deltas of observations (as an example, suppose I'm subtracting a student grade from another), should I normalize the delta of all students or just stick to the absolute delta?
3. Sticking to the same student case, If I have cumulative data (suppose for every test I sum their last grades). Should the process be the same?
4. Do we need to apply any data transformation like log or any other? If so, when should it be done? When data range is large? What's the point of changing the domain of the data here?
5. If I have a Categorical target, can I apply regression instead of classification so the output would be (suppose the classes are 0, 1, 2) 0.132, 0.431. Would it be more accurate?
6. In what kind of problems is Random Forest more indicated? Large datasets?
7. Should I discard the less important variables? Maybe it just creates noise?

I know the pre-processing depends on the problem, data, etc. and I know there are a lot more things to look for when pre-processing. Here I'm more trying to understand the concepts behind pre-processing the data and the key points to look for when doing so. So with that in mind, what would be the key points to look for when pre-processing data? (If I didn't mention any other important point, and I'm sure a lot is missing, please consider that too). Imagine you're teaching that to your grandpa :)

When doing predictions with Random Forests, we very often (or always) need to perform some pre-processing.

This is not true. Random Forest is really "off-the-shelf".

Outliers. Should we remove them all? If so, we consider an outlier based on the 3/2 rule? Should we keep them? Why?

The base model used in RF is a large decision tree (usually built via CART). Decision trees are robust to outliers, because they isolate them in small regions of the feature space. Then, since the prediction for each leaf is the average (for regression) or the majority class (for classification), being isolated in separate leaves, outliers won't influence the rest of the predictions (in the case of regression for instance, they would not impact the mean of the other leaves). Bottom line: you don't care about outliers in RF. Just remove them if they are aberrant observations (e.g., due to recording errors). If they're valid cases, you can keep them.

When dealing with deltas of observations (as an example, suppose I'm subtracting a student grade from another), should I normalize the delta of all students or just stick to the absolute delta? Sticking to the same student case, If I have cumulative data (suppose for every test I sum their last grades). Should the process be the same?

The question here is not really related to RF, it is algorithm independent. The real question is what do you want to do? What are you trying to predict?

Do we need to apply any data transformation like log or any other? If so, when should it be done? When data range is large? What's the point of changing the domain of the data here?

For the same reasons you don't need to worry about outliers, you don't need to apply any kind of data transformation when using RF. For classification, you may need to apply some kind of resampling/weighing strategy if you have a class-imbalance problem, but that's it.

If I have a categorical target, can I apply regression instead of classification so the output would be (suppose the classes are 0, 1, 2) 0.132, 0.431; so would it be more accurate?

You cannot apply regression if your target is categorical.

In what kind of problems is Random Forest more indicated? Large datasets?

RF is indicated for all types of problems. People (especially in the medical field, genomics, etc.) even use it primarily for its variable importance measures. In genetics, where the guys face the "small $n$ - large $p$" problem, RF also does very well. Anyhow, Machine Learning in general requires sufficient amounts of training and testing data, though there's no general rule. If your training data represents all your concepts and if these concepts are easily capturable, a couple of hundred observations may suffice. However, if what should be learned is very complex and if some concepts are underepresented, more training data will be needed.

Should I discart the less important variables? Maybe it just creates noise?

Another nice feature of decision trees built through CART is that they automatically put aside the non-important variables (only the best splitters are selected at each split). In the seminal book by Hastie et al. (2009), the authors showed that with 100 pure noise predictors, and 6 relevant predictors, the relevant variables were still selected 50% of the time at each split. So you really don't need to worry about variable selection in RF. Of course if you know that some variables are not contributing, don't include them, but if the underlying mechanisms of the process you're studying are mostly unknown, you can include all your candidate predictors.

• the comment below you seems to disagree stats.stackexchange.com/a/315218 with your statements (re outliers etc), tho they're upvoted similarly, so it's hard to know which to consider for someone who is unfamiliar. – baxx Oct 15 '20 at 22:54

Theoretically, Random Forest is ideal as it is commonly assumed and described by Breiman and Cuttler. In practice, it is very good but far from ideal. Therefore, these questions are very valid.

1. RF are not handling outliers as ideally as it is widely assumed. They are susceptible to even a single outlier with extreme values as was shown in How are Random Forests not sensitive to outliers?, and also there are couple of papers about how heteroscedasticity affects the RF predictions. In real-life data, you may have a lot (1-2%) of such outliers caused by typos (for human-inputted data like 3200 instead of 32.00), jumps of electrical current due to induction or simply due to unexpected exposures (for IoT), heteroscedasticity, etc. These "outliers" end up in many leaves of the decision trees, pulling predictions to higher values.

2. In case of unbalanced data where large number of target_value = 0, RF tends to underestimate predictions significantly.

3. Log-transformations can improve accuracy, especially in case of very skewed data (with very long tails). See for example "Forecasting Bike Sharing Demand" by Jayant Malani et al. (pdf), and this kaggle submission.

4. RF tends assigning higher importance to variables that have larger range of values (both categorical and continuous). For example, see this blog post: Are categorical variables getting lost in your random forests?

So, data preprocessing is very important even in the case of Random Forest.

I hope, this answer frames the validity of the questions and the links will provide some answers with starting points.

When pre-processing data, you are generally trying to achieve the following:

A. Removal of errors from your data. If your outliers are due to data recording errors, for example, you would want to fix this in the pre-processing stage. The various rules for identifying outliers should be treated as providing initial guesses, requiring further investigation.

B. Creating variables where you have a reasonable expectation that different values of the predictor variables may by correlated with the outcome variable. This is the bit that requires domain-specific knowledge, and good variables are often constructed using ratios, differences, averages of variables, etc.

C. Modifying the data to avoid restrictive assumptions of whatever model we are fitting.

The super-cool thing about tree-based methods, like random forests, is that they require much less effort in the type C pre-processing. In particular, normalizing, removing non-error-outliers, discarding variables, and log transformations, are not generally required. But, the cost of tree-based methods is that they are data hungry, so with smaller samples (e.g., less than 10,000 case), my experience is that often a glm is going to do a better job, but this brings in the type C processing, which after 25 years of building models I still find a challenge.