2
$\begingroup$

I'm currently trying to build a tennis prediction model. Unfortunately, I have some issues that I hope you could help me to handle.

I have 1110 examples of matches from the year 2013, with their outcomes and (which seems to me) relevant features. I trained a SVM (e1701 package, R) on this training set with a 10 fold cross validation. The model does pretty well: the average accuracy on the 10 fold cv is about 81% of correct predictions.

To further test the model, I tried a "manual CV": I kept a part of the training set out, so I have a test set. I trained the model on the training set, with 10 fold CV, then tested it on the test set. The accuracy is about 81% too. I repeated this process a lot of time, with different sizes of test set and different part of the data set to be the test set. The accuracy is always about 81%. Sounds good.

But here is the problem: I tested the model on some tournament from the year 2014. And the accuracy is pretty low: about 65%.

I don't understand why... In your opinion, where does that difference in accuracy comes from ? What can I do to handle this problem ? Does it exists some tools to overcome this kind of issues ?

Thank you for your help.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

Probably because an other variable plays a part in the outcome in the year 2014.

It is my understanding that predictive models have a limited lifespan.

For your model, both your training set and the data you're looking to predict are from the 2013 season. For instance, each tennis player is one year older, some players might be past their prime shape, and others might have been reaching it in 2014 or on their way to reach it in later years.

I think the difference in your prediction rate comes from those type of factors.

In the phone industry, a model from Sept 2014 with a 80% good prediction might fall to a 60% good prediction in Jan 2015 and lower the following months.

You need to add more variables into your model.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. In fact, my features consist of an average over a certain number of previous matches of the statistics of the two player. I guessed that would be sufficient to take the "effect of time" into account. Do I misunderstand the effect of such features ? i will try to think about others relevant features to anticipate the changes over the years. I'm curious (and I think it can be an enlightening example): In the phone industry, how do you handle such a problem ? $\endgroup$
    – Gabriel Ducrocq
    Commented Aug 14, 2015 at 13:06
  • $\begingroup$ For the phone models, we usually create a new model each time the accuracy starts to drop. $\endgroup$
    – Amandine FAURILLOU
    Commented Aug 14, 2015 at 13:25
  • $\begingroup$ As for the tennis players (I am not all that acquainted with their features), I think the training of the player comes into play as a factor ie has the physical preparation changed over the year, has the player changed trainers or even what amount of rest between the seasons did they get. An average does not take seasonality into account (first months of the year, after winter break, end of the season, begining of the season etc) $\endgroup$
    – Amandine FAURILLOU
    Commented Aug 14, 2015 at 13:32
0
$\begingroup$

First, one thing that comes to mind is that the tournament data is different - the time effect was already mentioned before. Other possibilities include differently encoded features or unusual conditions.

Another possibility is that you did both the cross validation and the evaluation on the test set incorrectly. I am slighty alarmed by

I have 1110 examples of matches from the year 2013, with their outcomes and (which seems to me) relevant features.

When testing the predictive power of your model it is important to not have the test data affect the model at all. One common and potentially very serious error is too pick features first using all the data. This includes an informal analysis using PCA or even just looking closely at the data.

Feature selection and hyperparameter tuning should always be done using training data only.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. "differently encoded features" Could you explain that ? In fact, I didn't performed any feature selection before using all the data. I had features at my disposal, and I used all of them. When I said they seemed relevant to me, I just meant they seemed meaningful, not that I chose them among others. So I don't think the test set affected the model this way. But maybe in some other way ? $\endgroup$
    – Gabriel Ducrocq
    Commented Aug 14, 2015 at 13:46
  • $\begingroup$ For your first question: the most standard problems are date and time formats. For example, time in seconds instead of minutes. Different missingness patterns. Things like that. As for the the second questions, it is difficult to tell without knowing what you did step-by-step. Feature selection is the most-common culprit, so this might not be the case here. $\endgroup$
    – Erik
    Commented Aug 14, 2015 at 13:51
  • $\begingroup$ In fact, my step-by-step procedure is very simple. I first tried the most brutal approach: I took all my features, trained a SVM on the training set, and then tested it on the test set. Without further brainstorming about the problem. And at my surprise, it worked very well on my data... $\endgroup$
    – Gabriel Ducrocq
    Commented Aug 14, 2015 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.