1
$\begingroup$

Background

I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the probability that the home team is going to win, throughout the game. I have implemented various machine learning algorithms and use various features (and combinations thereof).

Below is an image showing the win probability of the home team for two settings. Setting 1 uses two simple features while setting 2 uses the same two features plus four other features. The model I used in this case is Naive Bayes (with isotonic calibration). The four extra features seem relevant but ...

win probability of 2 settings

The problem

I don't know how to know/quantify whether using those four extra features has any impact on the quality of the predictions. This also applies for the other machine learning paradigms I used.

It is impossible to say that a series of predicted win probabilities for a game is accurate because the win probabilities are larger than 50% for most of the game while the home team won. As an example, let's say that the home team maintains a very big lead on the other team (in terms of points) for most of the game, but suddenly loses their lead at the end due to 2 injuries of their star players and thus ultimately losing. Let's also say that the predicted home team win probabilities are very high for most of the game, until the 2 injuries after which they drops a lot. In that case, I would say that the model is fairly accurate. But quantifying its accuracy by looking at which side the graph is most of the time, would deem the model very inaccurate (as it predicted that the home team would win most of the time).

I'm struggling to find any information about this problem on the internet. Does anybody have an idea about evaluating such a series of win probabilities or have some paper about this subject I could read?

$\endgroup$
  • $\begingroup$ This isn't really helpful, but: keep in mind that if you measure enough variables, you will have unique values for each game and your graph will tend to either constant 1 or constant 0. So, there has to be a balance re how many factors you use. $\endgroup$ – barrycarter May 1 '16 at 19:54
  • $\begingroup$ How do you measure that the 4 features are indeed relevant ? $\endgroup$ – Romain May 2 '16 at 17:54
  • $\begingroup$ @Romain, well that is exactly the problem. Intuitively, I think the 4 extra features are relevant as they add valuable information. But of course, I don't know how to measure that. $\endgroup$ – gieldops May 3 '16 at 9:54
2
$\begingroup$

Interesting problem! If I understand correctly you need to assess the quality of both models and take the one that works best: with better predictive power.

  1. Unless you have more games, the results will not be very generalizable. I assume that you have G games.

  2. You need to define a metric to compare the 2 models. When looking at binary outcome entropy (or log-loss) is usually an excellent metric. Then you can modify it to fit your purpose. The metric to score a model is always very subjective and especially in your case, you can define many. Here are a few examples:

    • Average Entropy (or log-loss) of the probability in the last 10 minutes. In that case, you are measuring how well a model predict "just" before the true answer is known: $$ Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{10}\sum_{i=40-10}^{i=40}{[y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}} $$

    • Average Entropy (or log-loss) of the probability with exponential loss. In that case, you are measuring how well a model predict "just" before the true answer is known: $$ Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{\sum_{i=1}^{i=M}{e^{i-M}}}\sum_{i=1}^{i=M}{e^{i-M} [y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}} $$

With $y_g$ being the outcome of the game (0/1). If you think that the beginning or the middle of the game is more important than you can adjust those function as required.

Hope this help.

$\endgroup$
  • $\begingroup$ Thanks for your answer! I will try to implement this in a few days and let you know of the results. $\endgroup$ – gieldops May 3 '16 at 9:55
  • $\begingroup$ how did this work for you? @gieldops $\endgroup$ – Tom Zinger Mar 12 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.