I have timesheet data that shows per employee of a company whenever he has taken a holiday combined with some other information such as his age etc. The observations start in 2001 and in 2016. Of course employees only remain observed if they have not yet left the company. So the data contains for every employee an observations for each day he has either worked or taken a holiday.
The goal is to predict in a given month how many holidays will be taken the month after, given the information we have in that particular month. So basically I aggregated all the data such that each month per year has 1 observation. This leaves us with 192 observations to train the model on (not that much, I know). The data look like this
HolidaysTakenInMonth HolidaysAlreadyTakenThisYear MonthName WorkingDays NoEmployees 351 451 August 20 200 421 521 September 18 210 100 621 October 19 215 ... ... ... ... 845 2541 September 18 615 655 2631 October 20 630 621 3212 November 21 730 ... ... ... ...
So I have 12 observations (one for each month) for each of the 16 years, which means I have 192 observations. The first variable is the target variable and I use to other variables to predict. It is notable here that the company has been growing, so each year and each month the number of employees increases and by consequency also the number of holidays taken in a particular month. But I assume I take this into account by using the variable 'NoEmployees', which represents the number of employees that work for the company at the beginning of each month.
I am a bit confused about how I should validate this model. Initially I used a 10-fold cross validation approach. This gave pretty good results: RMSE of 180 and R-squared of 0.96. But now I wonder whether 10-fold cross validation is a valid approach here. Even though I do not use a time-series approach (ARIMA e.g.) there is still a time-dimension that has the property of increasing employees each month. So I also looked at forward chaining. I first train on the year 2001 and test on 2002, then I train on 2001 and 2002 and test on 2003 and so on until in the last observation I train on 2001 until 2015 and test on 2016. This shows much more variation in the R-squared (for one year as low as 0.54, but for other years either somewhere > 0.80 or > 0.90).
So basically I have two questions:
Is 10-fold CV a valid approach here? If not, is forward chaining a valid approach?
What does the variation in R-squared and RMSE imply in my forward chaining approach?