I was a given a data set with different predictors about a store and the the idea is to forecast the number of daily shoppers. The predictors are: weekday, time of the day (morning, afternoon, evening), week number, month, weather (humidity, dew point, temperature), holidays. The outcome variable is the number of visitors.
I want to build a regression model to predict the number of visitors using traditional machine learning algorithms such as random forests, svm, and the like.
My main concern is how to validate this model using CV since some of the predictors are time related. Plain vanilla CV cannot be performed here. In this question, they suggest a way to perform this but my problem is that I only have data from June 2015 to present.
My initial idea was the following:
a. train with data from June 2015-December 2015. Test with January 2016. b. train with June 2015-January 2016. Test with February 2016. . . .
Each time one month of data is added to the training data after having asses the error for that month. Then compute average performance.
Is this approach reasonable or not?
If so, should I get rid of the month variable? Note that in a., for instance, I am testing with some data that belongs to different months that the one used for training. I mean, for the training I used data from June to December 2015, but I am testing for January 2016. Seasonality can be something I am missing.
How to validate such models in general?