I am using a multi-layer perceptron neural network to try to predict the outcomes of football matches, using 20 years worth of match results and statistics. I am using 10-fold cross validation in Weka, and getting results such as 83% correctly classified instances for my chosen inputs and parameters. I then train the model on the entire data set.
But when I apply the model to try to predict real world results of future matches (over the last 10 rounds), my model is only achieving an accuracy of 52%, barely better than chance.
In theory, if the cross validation results in 83% accuracy, shouldn't I expect the same model to achieve roughly the same accuracy in future predictions?
Edit: I also trained my network on 20 years worth of match data up to and including 2016 (i.e. excluding any 2017 data). Then I test the resulting model on a separate test set comprising 2017 match data, which was not used at all in the training, and the network achieves a 82% accuracy. And yet in practice, I have only achieved 52% prediction accuracy. I still don't understand that discrepancy.