4
$\begingroup$

currently I'm learning more about the Random Forest algorithm which I used together with 7 predictor variables (6 continuous and one binary factor variable) in order to predict patterns of international migration using a [publicly available dataset][1] on international (origin-destination) migration between 1960 and 2004 (where I use only those points from >1989). The dataset contains only non-zero migration, hence migration > 0. Of course the range of response-values is quite high (1-500.000) as is the variance.

This is what the summary of the response looks like.

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1      40     227    2394    1186  455100  

As predictors the author of the paper used population (origin, dest), area (origin, dest), distance between origin and dest (and several indicators which I left out).

Using RFSRC and RandomForest (R packages) likewise I was able to actually predict independent test data (not used for training, hence no OOB data) pretty well, when I was training the forest with data from 1989 - 1999 and compare predicted vs observed on 2000-2004 (actually the results looked MUCH more promising than the author's proposed model, three simple GLMs with and without interaction terms). I checked the time-series for various countries using the global model and of course, there were patterns missing due to the weak explanatory content of the predictors (war, financial crisis, etc.). Nevertheless, it could capture the magnitudes as well as the trends which is all I'm interested for now.

I have several questions:

  1. With regard to the metrics (summary):

                         Sample size: 15440
                     Number of trees: 50
          Minimum terminal node size: 5
       Average no. of terminal nodes: 3043.72
       No. of variables tried at each split: 5
              Total no. of variables: 7
                            Analysis: RF-R
                              Family: regr
                      Splitting rule: mse
                  variance explained: 75.77
                          Error rate: 25330627
    

Especially the error rate seems HORRIBLY high and I don't seem to be able to lower it using different parameter-sets. If I fit the model using the common logarithm on both the predictors and the response (like the author had to using his GLM!) than the summary looks different:

                         Sample size: 15440
                     Number of trees: 50
          Minimum terminal node size: 5
       Average no. of terminal nodes: 3043.72
 No. of variables tried at each split: 5
              Total no. of variables: 7
                            Analysis: RF-R
                              Family: regr
                      Splitting rule: mse
                  variance explained: 91.12
                          Error rate: 0.1

Now, since the response is no longer Poisson but normally distributed (min/max range: 0-5.658 log(migrants)/year) the summary looks better. For me this seems plausible since the transformation dampens the variance structure of the data. And indeed, observed vs. predicted on my independent test data set looks good on log-scale. But if I transform my results back and compare obs. vs. pred on the exponentiated data then the residuals are quite big.

My question is, how far I should care about the error rate? What does it tell me exactly and to what extent is random Forest sensitive to the scale of continous input variables?

  1. As people keep on saying, that RF is not able to project I wanted to ask which ML-algorithm is capable of predicting beyond the scope of training data?

As a side-note: I don't wish to discuss the quality of the author's covariate choice here. I'm aware, that the explanatory content of the models' results have to be taken with care. I mainly use his data because It's a complete compilation of origin-destination migration data ready to play with.

$\endgroup$
  • 1
    $\begingroup$ Interesting application. There could be many answers to odd results, a reproducible example always helps. The link is broken. $\endgroup$ – Soren Havelund Welling Feb 16 '16 at 17:51
2
$\begingroup$
  1. Typically no error-rate would be reported for regression, and the number should not be more than one. "Can't make the same mistake twice". Yes log transformation is likely to dampen error term. If your cost of wrong predictions happened to be directly proportional to migration, log transformation may only mask how bad sometimes the model behaves.

  2. Supervised regression does not have to project to predict future outcomes. It depends on what assumptions your're willing to make. You can train a RF model to predict the rate of change one defined period ahead with a rolling window.

$\endgroup$
  • $\begingroup$ Thanks for your answer. Could you elaborate a little more on why the error rate would be non-existent when using RF in a regression context? Also I don't quite understand what you mean by saying "Can't make the same mistake twice". Thanks! $\endgroup$ – A. Neumann Feb 19 '16 at 15:03
  • $\begingroup$ Regression models predict numbers. The error is the absolute residual between predicted value and true value. Regression error is e.g. summarized as mean_error, mean_squared_error or explained variance. I guess one could define error_rate for regression as % of times predictions were not entirely accurate. Not a very useful metric. Anyhow error_rate is wrong predictions divided by all predictions and cannot exceed 1. $\endgroup$ – Soren Havelund Welling Feb 19 '16 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.