Ultra-high error rate using Random Forest regression in R (international migration data)

Question

currently I'm learning more about the Random Forest algorithm which I used together with 7 predictor variables (6 continuous and one binary factor variable) in order to predict patterns of international migration using a [publicly available dataset][1] on international (origin-destination) migration between 1960 and 2004 (where I use only those points from >1989). The dataset contains only non-zero migration, hence migration > 0. Of course the range of response-values is quite high (1-500.000) as is the variance.

This is what the summary of the response looks like.

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1      40     227    2394    1186  455100  

As predictors the author of the paper used population (origin, dest), area (origin, dest), distance between origin and dest (and several indicators which I left out).

Using RFSRC and RandomForest (R packages) likewise I was able to actually predict independent test data (not used for training, hence no OOB data) pretty well, when I was training the forest with data from 1989 - 1999 and compare predicted vs observed on 2000-2004 (actually the results looked MUCH more promising than the author's proposed model, three simple GLMs with and without interaction terms). I checked the time-series for various countries using the global model and of course, there were patterns missing due to the weak explanatory content of the predictors (war, financial crisis, etc.). Nevertheless, it could capture the magnitudes as well as the trends which is all I'm interested for now.

I have several questions:

1. With regard to the metrics (summary):

                        Sample size: 15440
                    Number of trees: 50
         Minimum terminal node size: 5
      Average no. of terminal nodes: 3043.72
      No. of variables tried at each split: 5
             Total no. of variables: 7
                           Analysis: RF-R
                             Family: regr
                     Splitting rule: mse
                 variance explained: 75.77
                         Error rate: 25330627
   

Especially the error rate seems HORRIBLY high and I don't seem to be able to lower it using different parameter-sets. If I fit the model using the common logarithm on both the predictors and the response (like the author had to using his GLM!) than the summary looks different:

                         Sample size: 15440
                     Number of trees: 50
          Minimum terminal node size: 5
       Average no. of terminal nodes: 3043.72
 No. of variables tried at each split: 5
              Total no. of variables: 7
                            Analysis: RF-R
                              Family: regr
                      Splitting rule: mse
                  variance explained: 91.12
                          Error rate: 0.1

Now, since the response is no longer Poisson but normally distributed (min/max range: 0-5.658 log(migrants)/year) the summary looks better. For me this seems plausible since the transformation dampens the variance structure of the data. And indeed, observed vs. predicted on my independent test data set looks good on log-scale. But if I transform my results back and compare obs. vs. pred on the exponentiated data then the residuals are quite big.

My question is, how far I should care about the error rate? What does it tell me exactly and to what extent is random Forest sensitive to the scale of continous input variables?

2. As people keep on saying, that RF is not able to project I wanted to ask which ML-algorithm is capable of predicting beyond the scope of training data?

As a side-note: I don't wish to discuss the quality of the author's covariate choice here. I'm aware, that the explanatory content of the models' results have to be taken with care. I mainly use his data because It's a complete compilation of origin-destination migration data ready to play with.

Answer 1

1. Typically no error-rate would be reported for regression, and the number should not be more than one. "Can't make the same mistake twice". Yes log transformation is likely to dampen error term. If your cost of wrong predictions happened to be directly proportional to migration, log transformation may only mask how bad sometimes the model behaves.

2. Supervised regression does not have to project to predict future outcomes. It depends on what assumptions your're willing to make. You can train a RF model to predict the rate of change one defined period ahead with a rolling window.