I have some daily data from city A, B, C. Values from city A are highly correlated with values from other cities for lag -1,-2,-3 and -4. I want to use Random Forest, SVM and ANN to predict values for city A. My idea is:

  1. Split data into training and testing set.
  2. Use the formula: valueA ~ valueB-1 + valueB-2 + valueB-3 + valueB-4 + valueC-1 + valueC-2 + valueC-3 + valueC-4

  3. Try different methods (Random Forest, SVM and ANN) on training set and use createTimeSlices to cross-validation for model training and parameter tuning, like in this example - https://stackoverflow.com/a/22338029/2602477

  4. Evaluate obtained model on the testing set (using R2, RMSE, etc.)

My questions are:

  • How to properly split data into training and testing set? I'm not sure, but boostraping or k-fold cross-validation doesn't sound right.
  • Are given formula is correct for this case?

I'll take the alternative approach to @forecaster and suggest you have the option of not treating this as a time series problem. Instead, with A as the response, pre-compute predictor values for each of the lags. That is, before training, add a column for each lagged value to your data frame, say


and so on. That way you don't have to worry about keeping rows together during sampling and can use bootstrap and k-fold just fine. Your formula becomes A ~.. Depending on which learning technique you use you may find that certain "highly correlated" cities and lags are in fact not good predictors for A. Moreover if your predictors are highly correlated you can get rid of some. I'd suggest using the caret package to simplify your programming task, qualify your predictors, and evaluate your modeling results.

[Edit] I'm not clear why you're interested in bootstrapping. You can use the partition creator of caret to make splits easily. If you have plenty of data you may not need bootstrapping, but if you want resampling you can uses caret for that too (see createResample). Here is a template for how you might use caret to perform a random forest fit with your data. The formula shown is the A~. suggested above which assumes your response is A in the training and testing data frames.


# set up predictors and responses
predictors <- getPredictors(...) # your predictors from B, C, D, rows by time
responses <- getResponses(...) # your responses from A, rows by time

# examine

correlations <- cor(na.omit(predictors))

     title="Predictor Correlation (HC)",
     title="Predictor Correlation (First PC)",
     title="Predictor Correlation (Ang. Ord. Eigenvectors)",

# find highly-correlated predictors
complete <- predictors[complete.cases(predictors),]
predCor1 <- cor(complete)
highlyCorPred <- findCorrelation(predCor1, cutoff = 0.75)
filteredPred <- complete[, -highlyCorPred]
predCor2 <- cor(filteredPred)

# removed highly-correlated predictors

# find linear combinations

# using the filtered predictors, append the response then build partition 
clean.bin <- cbind(filteredPred,responses[index(filteredPred),]) # index from time series
clean.bin <- as.data.frame(clean.bin[complete.cases(clean.bin),]) # responses may have added NAs
colnames(clean.bin)[15] <- "A"

# build training and testing sets
inTrain <- createDataPartition(y=clean.bin$A,times=1,p=0.7,list=FALSE)
training <- clean.bin[inTrain,]
testing <- clean.bin[-inTrain,]

# setup learning method
require(parallel) # optional

# applies for each classification or regression fit
fitControl <- trainControl(
  method = "repeatedcv",
  number = 10,
  repeats = 10,
  classProbs = TRUE,
  verboseIter = TRUE,
  seeds = NA,
  allowParallel = TRUE

# try the random forest fit
# using parallel computation if available
rfGrid = expand.grid(mtry = c(10,20,40,80))
cluster <- makeCluster(detectCores())
fit.raf <- train(A~.,
             tuneGrid = rfGrid,
             ntree = 1000,

predicted.raf <- predict(fit.raf,newdata=testing)

The caret documentation is excellent (e.g. caret model training) so you can explore many other options there. You might also use the time series split of caret ?createTimeSlices to use the techniques suggested by @forecaster.

| cite | improve this answer | |
  • $\begingroup$ I'd done something like this. Firstly, I prepared my data, the same as in your example. And my problem is (was?) how to do proper model training and evaluation on this data. I wasn't sure that I can use bootstrap for data split. $\endgroup$ – Jot eN Aug 1 '14 at 18:51
  • $\begingroup$ Edited post to add a template study. You can do a lot of things with time series analysis as others have posted. You can do a lot of things without time series analysis too. Caret will be useful to you and you can read a lot more about the techniques and the package in Kuhn & Johnson's "Applied Predictive Modeling". $\endgroup$ – mrbcuda Aug 1 '14 at 21:09
  • $\begingroup$ Just to clarify, as noted in my post I have stressed in 3 different times that the OP question is NOT doing a time series regression. so taking the rf/nn/svm route would work here.There is not time in Y/dependent variable. So any data dredging methods would work. $\endgroup$ – forecaster Aug 1 '14 at 23:15
  • 2
    $\begingroup$ In addition, just because of using a lag variable doesn't mean that we could shuffle the data, especially if you are predicting future values, t-1 has to happen before t happens, so it should be serielly ordered. I have yet to see a data mining book that covers a time series/dimension problem may be, RF/NN/SVM doesn't work well on time series problems. None of the techniques have been empirically tested on time series problems. $\endgroup$ – forecaster Aug 1 '14 at 23:16
  • 1
    $\begingroup$ Function embed allows obtaining lagged values for multiple lags, keeping variable length the same for each lag - that could be useful. $\endgroup$ – Richard Hardy Nov 26 '14 at 9:34

Have a look at the ARIMAX model specification. It seems to be closest to what you're doing.

You can use any nonparametric regression or classification model given concatenated Toeplitz design matrices. There's a slight issue, in that your models might not be stable in the sense outlined in this paper: Why Yule-Walker Should Not Be Used For Autoregressive Modelling

With forecasting time-series models, you want the data to be preprocessed so that it has approximately the same mean and standard deviation everywhere. Depending on your data, you probably want to preprocess your data by differencing, in which you subtract each value from the previous. For forecasting purposes, the inverse of this operation is the cumulative sum plus your burnt in value at the beginning. This operation can be applied multiple times to make your data appear stationary. Another preprocessing option is to perform low order polynomial regression on the data, then predict the residuals.

The appropriate form of cross validation for time-series to take the first 60% as your training set, the next 20% as your validation set and the final 20% as your test set.

If you have enough data you can restrict yourself to online (iterative) models, you can have an efficient estimate of performance. A neat trick for with confidence bands in online time series models was presented in this paper: Online Reliability Estimates for Individual Predictions in Data Streams.

If you want a stronger background in the theory of time-series, I cannot recommend Brockwell and Davis enough. Unfortunately, there's no legal free source of the PDF.

| cite | improve this answer | |

At this moment, there is no 'rule' as how to divide a data set into a training, validation and testing set. However, 80/20 (Pareto) is a popular and widely accepted way of dividing your data set. This translates into 80% training(1) and 20% testing. That 80% training(1) is then divided into training(2) and validation data in the same way (80/20). That's about the same as the 60/20/20 that @Jacob Mick suggested in the other answer (and perhaps more straight forward).

As you mentioned a time series, I wouldn't use k-fold, but rather a holdout sample for your validation and testing (aka 1-fold). More specifically, the last part of your data. Otherwise you'll be going/predicting back in time and forth during optimization. Something I would avoid.

For your formula...I really don't have enough information about the values to tell you whether it's a good formula in this case.

| cite | improve this answer | |

Below are my responses:

You could do K fold cross validation/ leave one out cross validation as follows for time series regression: if you have 100 time series observation, first determine how many minimum# of observation is required to build your model.

  1. Lets say you need at least observations 1 to 50 to build your model. Build a model on the first 50 observations.
  2. Test your model on the next n (say 10 observations).
  3. Now rebuild the model on the 50 + 10 (n) = 60 time ordered data. Test the model on 61 to 70 observations.
  4. Repeat the steps 1 to 3 by increment the training dataset by 10 data points. If you did this and calculate the prediction error you would have done the k (5) fold validation of your model.

It is very important to note that the data should be time ordered and should not be shuffled or selected randomly. In the example above, observation 1 would be your earliest time data, while your 100th observation would be your last data point, sorted in an ascending order so that your lag structure in the model is not disturbed.

A variation of the above approach would be, if you only want 50 observations in your training data to build model, you need to drop the first 10 observation as you keep adding l0 observations at last, as an example:

  • Step 1: Train on 1 to 50 observations, test on 51 to 60.
  • Step 2: Train on 11 to 60, test on 61 to 70.
  • .... and so on,

This could be easily expanded to leave one out cross validation.

Here is the problem you might face, while you have nicely arranged the train and test sets, machine learning methods, randomly shuffle the data to build models. Example, random forest will not care if it is time ordered or not. it will randomly shuffle the dataset to build models. At the end it doesn't matter if you use a time series cross validation or a regular cross validation, as long as your residuals of the model is not correlated.

In your case I think it will not be autocorrelated unless if you have different time periods for a single city in you dependent variable as an example:

$valueA_t = valueB-1 + valueB-2 + valueB-3 + valueB-4 + valueC-1 + valueC-2 + valueC-3 + valueC-4$

where $t = time$ The key is suffix $t$ in your dependent variable. If you have time dependency in your dependent variable the you have to do a time series cross validation.

But based on your problem statement, your are NOT doing a time series regression, since your dependent variable doesn't have time dependency/serial dependance on previous observation i.e obs 2 (time 1) is dependent on obs 1 (time 2), and they are independent of each other, so you should be fine using a regular cross validation.

See the following link on how to do proper cross validation for time series data for forecasting. http://robjhyndman.com/hyndsight/tscvexample/

In case you are in fact doing a time series regression, then I would be using tradition statistical based time series models such as ARIMAX models or regression with arima errors because, machine learning methods are notoroiusly know to perform extremely poorly on time series data and there is no known empirical evidence that using random forest/neural network is known to improve prediction. As shown in the neural network forecasting competition, statistical time series methods significantly out performed neural networks. http://www.neural-forecasting-competition.com/NN3/results.htm

Again in your case, I dont think you are doing a time series regression, so you should be fine using machine learning methods.

| cite | improve this answer | |
  • 1
    $\begingroup$ I've had luck in the past with time-series regression. It is sensitive to preprocessing via differencing, detrending and removing seasonality. For most problems I've encountered it works better than linear models. For a primer see: ulb.ac.be/di/map/gbonte/ftp/time_ser.pdf $\endgroup$ – Jessica Collins Jul 29 '14 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.