7
$\begingroup$

I am comparing a random forest model to a GLS model using a univariate time series that has a deterministic linear trend. I am going to add a linear time trend covariate (among other predictors) to the GLS model to account for the changing trend. To be consistent in my comparison, I was hoping to add this predictor to the random forest regression model as well. I have been looking for literature on this subject and can't find much.

Does anyone know if adding this type of predictor is inappropriate in a random forest regression for any reason? The random forest regression already includes time-lagged variables to account for autocorrelation.

$\endgroup$
9
$\begingroup$

RFs, of course, can identify and model a long-term trend in the data. However, the issue becomes more complicated when you are trying to forecast out to never seen before values, as you often are trying to do with time-series data. For example, if see that activity increases linearly over a period between 1915 and 2015, you would expect it to continue to do so in the future. RF, however, would not make that forecast. It would forecast all future variables to have the same activity as 2015.

from sklearn import ensemble
import numpy as np
years = np.arange(1916, 2016)
#the final year in the training data set is 2015
years = [[x] for x in years]
print 'Final year is %s ' %years[-1][0]
#say your ts goes up by 1 each year - a perfect linear trend
ts = np.arange(1,101)
est = ensemble.RandomForestClassifier().fit(years,ts)
print est.predict([[2013], [2014], [2015], [2016] , [2017], [2018]])

The above script will print 2013, 2014, 2015, 2015, 2015, 2015. Adding lag variables into the RF does not help in this regard. So careful. I'm not sure if adding trend data to your RF is gonna do what you think it will.

$\endgroup$
  • 6
    $\begingroup$ This is an excellent point, thank you for pointing this out! The problem is that the RF will split on the trend predictor, but of course any future trend values will simply be treated as the last split bucket, not be extrapolated. The solution would be to use the trend predictor in a simple linear regression in the leaves of the trees. Unfortunately, it seems like randomForest() in R does not allow specifying this. Does numpy? If not, the OP would need to implement her own RF, based on tree implementations that do allow regressions in the leaves. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 8 '15 at 6:54
  • 3
    $\begingroup$ It seems like the cubist package for R allows regressions in the tree leaves, so one could implement an RF around this. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 8 '15 at 7:09
  • $\begingroup$ Interesting! I hadn't considered that. Cubist seems like a great package. Unfortunately, I can't find a python proxy and, if others are interested, Cubist does not work on Macs. $\endgroup$ – captain_ahab Oct 8 '15 at 18:12
  • $\begingroup$ All great information! Thank-you both for the discussion. I am actually not focusing on the forecasting abilities for this particular project, but rather a comparison of different model fits to many different simulated data sets (one including a linear trend). I really only want to add the linear time trend to the RF to be consistent with the trend variables that I am using in the other regression models. But I do have a question about the forecasting abilities. In general, are RFs poor forecasters for data with trends? Even with or without adding a time trend variable? $\endgroup$ – Hannah Oct 8 '15 at 18:48
  • $\begingroup$ I think that RFs that actually regress on the trend would be quite competitive. (As discussed here, simply splitting on the trend won't extrapolate it into the future.) As discussed, RFs would automatically dampen such a regressed trend, which would make a lot of sense. However, I personally haven't come across a convincing study of forecasting trended data with RFs in the literature. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 8 '15 at 21:42
5
$\begingroup$

Just change the variable you are trying to predict to the difference in the dependent variable.

As the other posts point out, the random forest will not know how to treat time variables that occur after the training set. Let's say your training set has data from Minute 1 to Minute 60. The random forest might make a rule that after forty minutes the dependent variable is 100. Even if there is a trend, if you get out to Minute 10000 in the test data, the same rule will be applied. If you predict the difference though, this can have the same effect of including a trend.

As to whether RF's are decent forecasters, I have had MUCH greater luck with RF's than other econometric models like VAR, VECM, etc. but especially for short-term forecasts. Some other models do seem to work better on most data, however, such as well-tuned GBM models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.