7
$\begingroup$

This is a simple question, is it okay to use a Random Forest model on Time Series Data? I ask this because, in a random forest model, we perform bootstrapping of observations where we randomly sample from the training set with replacement. Doesn't this ruin the "ordering of observations" in the model, since it being time series data. I'm asking this with context to financial data, say I'm doing a classification type problem to buy/not buy an asset, and I collect daily data for some features to predict this variable.

$\endgroup$
6
$\begingroup$

It works well but only if the features are properly prepared so that the order of the lines is not important anymore.

E.g. for a univariate time series $y_i$, you would use $y_i$ as response and e.g. the following features:

  1. Lagged versions $y_{i-1}$, $y_{i-2}$, $y_{i-3}$ etc.

  2. Differences of appropriate order, e.g. $y_{i-1} - y_{i-2}$, $y_{i-1} - y_{i-8}$ (if there is weekly seasonality expected and the observations occur daily) etc.

  3. Integer or dummy coded periodic time info such as month in year, week day, hour of day, minute in hour etc.

The same approach works for different modelling techniques, including linear regression, neural nets, boosted trees etc.

An example is the following (using a binary target "temperature increase" (y/n)):

library(tidyverse)
library(lubridate)
library(ranger)
library(MetricsWeighted) # AUC

# Import
raw <- read.csv("https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv")

# Explore
str(raw)
head(raw)
summary(raw)
hist(raw$Temp, breaks = "FD")

# Prepare and add binary response
prep <- raw %>% 
  mutate(Date = ymd(Date),
         y = year(Date),
         m = month(Date),
         d = day(Date),
         increase = 0 + (Temp > lag(Temp)))

with(prep, table(y))
summary(prep)

# Plot full data -> year as seasonality
ggplot(data = prep, aes(x = Date, y = Temp))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

# No visible within year seasonality
prep %>% 
  filter(y == 1987) %>% 
ggplot(aes(x = Date, y = Temp))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

# Add some lags and diffs & remove incomplete rows
prep <- prep %>% 
  mutate(lag1 = lag(Temp),
         lag2 = lag(Temp, 2L),
         lag3 = lag(Temp, 3L),
         dif1 = lag1 - lag2,
         dif2 = lag2 - lag3) %>% 
  filter(complete.cases(.))

# Train/valid split in blocks
valid <- prep %>% 
  filter(y == 1990)
train <- prep %>% 
  filter(y < 1990)

# Models
y <- "increase" # response
x <- c("lag1", "lag2", "lag3", "dif1", "dif2", "y", "m", "d") # covariables
form <- reformulate(x, y)

# Logistic model: Linear dependence between difs and lags
fit_glm <- glm(form, 
               data = train, 
               family = binomial()) 
summary(fit_glm)

# Random forest
fit_rf <- ranger(form, 
                 data = train,
                 seed = 345345, 
                 importance = "impurity", 
                 probability = TRUE)
fit_rf
barplot(-sort(-importance(fit_rf))) # Variable importance

# Evaluate on 1990 for glm by looking at ROC AUC
pred_glm <- predict(fit_glm, valid, type = "response")
AUC(valid[[y]], pred_glm) # 0.684 ROC AUC

# Then for rf
pred_rf <- predict(fit_rf, valid)$predictions[, 2]
AUC(valid[[y]], pred_rf)    # 0.702 ROC AUC

# view OOB residuals of rf within one month to see if structure is left over
random_month <- train %>% 
  mutate(residuals = increase - fit_rf$predictions[, 2]) %>% 
  filter(y == 1987, m == 3) 

ggplot(random_month, aes(x = Date, y = residuals))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

Replacing variables "y" and "m" by factors would probably improve the logistic regression. But since the question was about random forests, I leave this to the reader.

$\endgroup$
  • 1
    $\begingroup$ "but only if the features are properly prepared so that the order of the lines is not important anymore." If I understand you correctly, you basically need to calculate all the lagged versions to be features, so the order of the lines is no longer important? $\endgroup$ – Amonet Jul 18 at 7:49
  • 2
    $\begingroup$ Yes exactly: Lags, differences etc. $\endgroup$ – Michael M Jul 18 at 10:36
  • $\begingroup$ @MichaelM - I have two quick questions; when entering data for lagged versions, what do you do when there is not a value (for example, y_i-2 for y_i)? Would it be a NaN, 0...? Also, is there any place where I could find an example of how to handle the differences? I am not sure I understand what I am trying to do there. $\endgroup$ – Irina Sep 13 at 16:23
  • $\begingroup$ I'd probably remove the few lines without lag. By differences I mean those between different lagged versions of the time series. So e.g. if you have hourly values $y_{i}$ of river temperatures, you can use $y_{i-1}$, $y_{i-2}$, $y_{i-24}$, $y_{i-1} - y_{i-24}$ and $y_{i-1} - y_{i-2}$ as predictors. This example is for a univariate series. $\endgroup$ – Michael M Sep 13 at 17:12
  • 1
    $\begingroup$ I have tried to add this example with binary response. $\endgroup$ – Michael M Sep 16 at 18:36
2
$\begingroup$

A random forest would not be expected to perform well on time series data for a variety of reasons. In my view the greatest pitfalls are unrelated to the bootstrapping, however, and are not unique to random forests:

  1. Time series have an interdependence between observations, which the model will ignore.
  2. The underlying learner is typically a tree based algorithm, which does not extrapolate trends. If there are genuine time trends in the data these will not be projected forward.
$\endgroup$
  • 2
    $\begingroup$ Random Forest can be, and is used for time-series predictions. Look at a few examples: Dudek, G. (2015). Short-term load forecasting using random forests. In Intelligent Systems' 2014 (pp. 821-828). Springer, Cham./// Mei, J., He, D., Harley, R., Habetler, T., & Qu, G. (2014, July). A random forest method for real-time price forecasting in New York electricity market. In 2014 IEEE PES General Meeting| Conference & Exposition (pp. 1-5). IEEE. $\endgroup$ – Shahin May 25 at 12:27
  • $\begingroup$ @MichaelM how does RF deal with features like multiple level shifts and/or multiple deterministic time trends. Does one have to specify them. How about the window of response around holidays with each holiday having its own pattern ? $\endgroup$ – IrishStat Sep 16 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.