# Cross-validating data with (aggregated) time dimension

I have timesheet data that shows per employee of a company whenever he has taken a holiday combined with some other information such as his age etc. The observations start in 2001 and in 2016. Of course employees only remain observed if they have not yet left the company. So the data contains for every employee an observations for each day he has either worked or taken a holiday.

The goal is to predict in a given month how many holidays will be taken the month after, given the information we have in that particular month. So basically I aggregated all the data such that each month per year has 1 observation. This leaves us with 192 observations to train the model on (not that much, I know). The data look like this

HolidaysTakenInMonth HolidaysAlreadyTakenThisYear MonthName WorkingDays NoEmployees
351                  451                          August    20          200
421                  521                          September 18          210
100                  621                          October   19          215
...                  ...                          ...       ...
845                  2541                         September 18          615
655                  2631                         October   20          630
621                  3212                         November  21          730
...                  ...                          ...       ...


So I have 12 observations (one for each month) for each of the 16 years, which means I have 192 observations. The first variable is the target variable and I use to other variables to predict. It is notable here that the company has been growing, so each year and each month the number of employees increases and by consequency also the number of holidays taken in a particular month. But I assume I take this into account by using the variable 'NoEmployees', which represents the number of employees that work for the company at the beginning of each month.

I am a bit confused about how I should validate this model. Initially I used a 10-fold cross validation approach. This gave pretty good results: RMSE of 180 and R-squared of 0.96. But now I wonder whether 10-fold cross validation is a valid approach here. Even though I do not use a time-series approach (ARIMA e.g.) there is still a time-dimension that has the property of increasing employees each month. So I also looked at forward chaining. I first train on the year 2001 and test on 2002, then I train on 2001 and 2002 and test on 2003 and so on until in the last observation I train on 2001 until 2015 and test on 2016. This shows much more variation in the R-squared (for one year as low as 0.54, but for other years either somewhere > 0.80 or > 0.90).

So basically I have two questions:

1. Is 10-fold CV a valid approach here? If not, is forward chaining a valid approach?

2. What does the variation in R-squared and RMSE imply in my forward chaining approach?

1. Either cross validation or forward chaining is a valid approach. (Incidentally, what you call "forward chaining" is often called "rolling origin out-of-sample testing" or "rolling holdout testing" or similar in the time series literature.)

2. I see why you could be surprised by the increase in RMSE over time. I don't think it is an artifact of your forward chaining approach. Test this: model the first half of your data using CV and predict the next year, and I strongly suspect you will get a lower RMSE than if you model all-but-the-last-year and predict the last year.

The problem is that now only will your mean number of days taken off increase with the number of employees, but the variance will also increase. If you have 10 employees, all of which can take anywhere from 0 to 10 days off and will take an average of 7 days each, then the mean total will be 70 with a possible range of 0-100. If you increase your number of employees to 20, then the mean will increase to 140, but the possible range will now go to 0-200. And of course the RMSE depends on the variance of the target variable, so it will increase with a larger variance.

If all you are interested in is the mean response, then you now know why the RMSE will increase, but it shouldn't be much of a concern. Alternatively, you could model the number of days taken off per employee. There, you will get the opposite effect: the variance will go down as the number of employees increases, because you get averaging effects. Yet another alternative would be to model first differences, i.e., the increase in total days taken off year-over-year. This would be especially appropriate if your employee growth is linear, because then this increase in days taken off simply reflects the days taken off by net new employees.

• Thank you. I understand that when there are more employees the variance will be higher (and accordingly also the RMSE). However, there is also fluctuation in the R-squared, which by design should correct for the increase in variance. Oct 14, 2017 at 11:04
• $R^2$ will of course fluctuate if you use different data sets. Does $R^2$ systematically decrease as you extend your training data? Oct 14, 2017 at 11:08

So basically I aggregated all the data such that each month per year has 1 observation... Initially I used a 10-fold cross validation approach.

Asides from what you asked, it looks like your aggregation is not fully utilizing the data available to you. For example:

1. Suppose there's an employee who's taken days off on the same month for the last 15 years. It is more probable that they s/he will take off on that month next year, than an employee who just joined this year.

2. Age might be correlated with family status (e.g., number of children), and that might be correlated with the holiday choices (e.g., whether it corresponds to school holidays).

3. The number of days taken off by the employee before December of some year, e.g., might be a good indication of the number of days taken off in December of that year.

So instead, perhaps you could organize the data differently. Create 12 datasets, one per month. For each month:

1. Create a table with an instance per employee. It should correspond to the last year you have.

2. The response variable(s) will be the number of vacation days per this month taken by the employee in the final year.

3. The features will be the age, the average number of vacation days per month by the employee in the preceding years, the number of years affecting this average, and the number of vacation days already taken off by the employee this year.

Now, for each month-dataset, perform the cross validation across employees. This should give you a prediction per employee. You should now divide the variance by the number of employees (under the assumption that the errors are independent).

Note that you will have different performances per month, which intuitively makes sense. The number of days in a month including Christmas, for examples, might be easier to predict than some other month.

• Those are some interesting points, thank you. But note that I also include a variable 'MonthName' in my regression, which is a factor variable representing the 12 months. So for a month including Christmas the prediction will be higher due to this variable. Also, several employees will start e.g. in 2017. So for these employees we don't have observations on their previous behavior and for many others there will be very limited information. So on the one side you might gain information by predicting per employee, but I think that on the other side you might also lose some info. Oct 14, 2017 at 11:02