I'm currently trying to build a tennis prediction model. Unfortunately, I have some issues that I hope you could help me to handle.
I have 1110 examples of matches from the year 2013, with their outcomes and (which seems to me) relevant features. I trained a SVM (e1701 package, R) on this training set with a 10 fold cross validation. The model does pretty well: the average accuracy on the 10 fold cv is about 81% of correct predictions.
To further test the model, I tried a "manual CV": I kept a part of the training set out, so I have a test set. I trained the model on the training set, with 10 fold CV, then tested it on the test set. The accuracy is about 81% too. I repeated this process a lot of time, with different sizes of test set and different part of the data set to be the test set. The accuracy is always about 81%. Sounds good.
But here is the problem: I tested the model on some tournament from the year 2014. And the accuracy is pretty low: about 65%.
I don't understand why... In your opinion, where does that difference in accuracy comes from ? What can I do to handle this problem ? Does it exists some tools to overcome this kind of issues ?
Thank you for your help.