1
$\begingroup$

I am using a multi-layer perceptron neural network to try to predict the outcomes of football matches, using 20 years worth of match results and statistics. I am using 10-fold cross validation in Weka, and getting results such as 83% correctly classified instances for my chosen inputs and parameters. I then train the model on the entire data set.

But when I apply the model to try to predict real world results of future matches (over the last 10 rounds), my model is only achieving an accuracy of 52%, barely better than chance.

In theory, if the cross validation results in 83% accuracy, shouldn't I expect the same model to achieve roughly the same accuracy in future predictions?

Edit: I also trained my network on 20 years worth of match data up to and including 2016 (i.e. excluding any 2017 data). Then I test the resulting model on a separate test set comprising 2017 match data, which was not used at all in the training, and the network achieves a 82% accuracy. And yet in practice, I have only achieved 52% prediction accuracy. I still don't understand that discrepancy.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Short answer: overfitting. $\endgroup$
    – Firebug
    Commented Jul 23, 2017 at 0:13
  • 2
    $\begingroup$ If you're using cross validation to tune hyperparameters, then you'd need a second, outer cross validation loop to estimate performance. It's also necessary to make sure that absolutely no information is bleeding across CV folds, which can happen in subtle ways (e.g. even in preprocessing). It's possible that future data doesn't follow the same distribution that your training set does. $\endgroup$
    – user20160
    Commented Jul 23, 2017 at 1:00
  • $\begingroup$ @Firebug Isn't cross validation supposed to be the answer to overfitting? I.e. if you use cross validation, you're by definition not overfitting, right? $\endgroup$
    – AndrWeisR
    Commented Jul 23, 2017 at 5:56
  • $\begingroup$ @user20160 I already have my hyperparameters determined, and I'm now looking at cross validation to try to get a measure of the network's predictive ability. I don't think I'd have any information bleeding between folds. Each instance in the data should be independent of the others; they're not time-dependent. Maybe the future data doesn't follow the same distribution, but given that I'm looking at football matches, why would this year be any different to the last 20 years, in the absence of any significant rule changes etc? $\endgroup$
    – AndrWeisR
    Commented Jul 23, 2017 at 6:07
  • 1
    $\begingroup$ "I don't think I'd have any information bleeding between folds." this is where you are most likely wrong: the huge discrepancy is typically a symptom of such dependence. And correlations along time series of data are to be expected. $\endgroup$
    – cbeleites
    Commented Jul 25, 2017 at 10:15

2 Answers 2

Reset to default
5
$\begingroup$

You are using cross-validation wrong. You just split all your data into 10 folds, which means in the training fold, there are events from every year, so it learns with 90% of the data of each year. When it predicts, it predicts the remaining 10% of a year from which it has already seen 90%. When you then predict future games, the classifier has not yet seen any data from that year (or course).

Or think of it like this: you want to predict the temperature of a certain day for the next year. If you use data from the last 20 years and split it, it is easy of course for the classifier to predict day x in the test sample if it has already seen x-1, x-2, x+1 (the days around day x). So it learns just to predict the next/past few days. It does newer learn to predict the next year, say to use the previous years to infer on the temperature on the same day a year later. The days before/after day x are way more useful.

I think as the problem is clear now, let's go to the solution.

So how to do unbiased cross-validation on time series:

  • do NOT use the same year in testing as well as training.
  • You also don't want to use newer data as the one you will predict, because this is not a real-world case...
  • take, let's say, the first 10 years, train, and predict year 11. Then train on the first 11 years, predict on 12 and so on. This will give you an estimation of how well the model is able to predict outcomes as well as how much it improves with more data. In the end, when you optimized your network, you can train on the full data sample.

This should yield more realistic performances and help you fight the current over-fit in your model.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. It's food for thought, although my data is not explicitly time-dependent. I don't have year, date or round number in my data. There is an implicit time dependence, in that a team's position on the ladder depends on the matches played to that point in the year. But I will try a sliding 10 year training set, evaluated on entire the following year. $\endgroup$
    – AndrWeisR
    Commented Jul 24, 2017 at 21:34
  • $\begingroup$ You're welcome. If you can, I think it is crucial to only predict games after your training data, this can also be the day after (so instead of splitting by years, train on nearly every year instead of the last couple of days of the last year, say.) in order to get a good evaluation of your classifier. $\endgroup$
    – Mayou36
    Commented Jul 25, 2017 at 7:56
  • $\begingroup$ Well, I tried training with data from 1997 to 2007, to predict 2008. Then 1997 to 2008 to predict 2009, 1997-2009 to predict 2010 etc, right up to 1997-2016 to predict 2017. The resultant accuracy was 83%, no different to my original 10-fold cross validation. $\endgroup$
    – AndrWeisR
    Commented Jul 28, 2017 at 11:53
  • $\begingroup$ Are you sure? 80% is quite large but possible, depending on what you exactly predict (if only win/tie/loose seems realistic). But then there is some significant difference between your 2017 dataset and the rest. Probably mixed up something? Or do you really say that you would have been able to predict the outcomes of the football matches of the last 20 years with 80% accuracy but in 2017 it dropped to 52%? This drop has to come from some corrupt data or similar thing (if you did the test mentioned above right with the yearly splits) $\endgroup$
    – Mayou36
    Commented Jul 28, 2017 at 14:19
0
$\begingroup$

I found my problem (for future reference). One of my input parameters was calculated, for training data, by looking at a statistical measure over all of the year's rounds played to date, but it included the current round itself. When predicting the future, the measure in question is not available for the current round. It is only available after the round has been played. My fix was to correct this input parameter to look at all rounds played to date, excluding the current round.

Once I made this change, my predicted accuracy dropped from 85% to a more realistic 70%, which is now more in accordance with observed predictions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's nice that you have solved the problem. But I think it would be nice if you would mark my answer as correct, as I think you are using what I have described above, right? The problem with the variable was more of an additional one if I understand you correctly. And basically what I suspected in my comment :) $\endgroup$
    – Mayou36
    Commented Aug 24, 2017 at 10:09
  • $\begingroup$ The problem didn't have anything to do with the way I was using cross validation, or dividing the data into yearly folds and predicting the next year, and so on. The problem was rather in how I was calculating one of the input parameters to the network, rather than how I was training it. $\endgroup$
    – AndrWeisR
    Commented Aug 25, 2017 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.