currently I'm learning more about the Random Forest algorithm which I used together with 7 predictor variables (6 continuous and one binary factor variable) in order to predict patterns of international migration using a [publicly available dataset] on international (origin-destination) migration between 1960 and 2004 (where I use only those points from >1989). The dataset contains only non-zero migration, hence migration > 0. Of course the range of response-values is quite high (1-500.000) as is the variance.
This is what the summary of the response looks like.
Min. 1st Qu. Median Mean 3rd Qu. Max. 1 40 227 2394 1186 455100
As predictors the author of the paper used population (origin, dest), area (origin, dest), distance between origin and dest (and several indicators which I left out).
Using RFSRC and RandomForest (R packages) likewise I was able to actually predict independent test data (not used for training, hence no OOB data) pretty well, when I was training the forest with data from 1989 - 1999 and compare predicted vs observed on 2000-2004 (actually the results looked MUCH more promising than the author's proposed model, three simple GLMs with and without interaction terms). I checked the time-series for various countries using the global model and of course, there were patterns missing due to the weak explanatory content of the predictors (war, financial crisis, etc.). Nevertheless, it could capture the magnitudes as well as the trends which is all I'm interested for now.
I have several questions:
With regard to the metrics (summary):
Sample size: 15440 Number of trees: 50 Minimum terminal node size: 5 Average no. of terminal nodes: 3043.72 No. of variables tried at each split: 5 Total no. of variables: 7 Analysis: RF-R Family: regr Splitting rule: mse variance explained: 75.77 Error rate: 25330627
Especially the error rate seems HORRIBLY high and I don't seem to be able to lower it using different parameter-sets. If I fit the model using the common logarithm on both the predictors and the response (like the author had to using his GLM!) than the summary looks different:
Sample size: 15440 Number of trees: 50 Minimum terminal node size: 5 Average no. of terminal nodes: 3043.72 No. of variables tried at each split: 5 Total no. of variables: 7 Analysis: RF-R Family: regr Splitting rule: mse variance explained: 91.12 Error rate: 0.1
Now, since the response is no longer Poisson but normally distributed (min/max range: 0-5.658 log(migrants)/year) the summary looks better. For me this seems plausible since the transformation dampens the variance structure of the data. And indeed, observed vs. predicted on my independent test data set looks good on log-scale. But if I transform my results back and compare obs. vs. pred on the exponentiated data then the residuals are quite big.
My question is, how far I should care about the error rate? What does it tell me exactly and to what extent is random Forest sensitive to the scale of continous input variables?
- As people keep on saying, that RF is not able to project I wanted to ask which ML-algorithm is capable of predicting beyond the scope of training data?
As a side-note: I don't wish to discuss the quality of the author's covariate choice here. I'm aware, that the explanatory content of the models' results have to be taken with care. I mainly use his data because It's a complete compilation of origin-destination migration data ready to play with.