Comparing and evaluating win probabilities in sports from different settings

Background

I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the probability that the home team is going to win, throughout the game. I have implemented various machine learning algorithms and use various features (and combinations thereof).

Below is an image showing the win probability of the home team for two settings. Setting 1 uses two simple features while setting 2 uses the same two features plus four other features. The model I used in this case is Naive Bayes (with isotonic calibration). The four extra features seem relevant but ...

The problem

I don't know how to know/quantify whether using those four extra features has any impact on the quality of the predictions. This also applies for the other machine learning paradigms I used.

It is impossible to say that a series of predicted win probabilities for a game is accurate because the win probabilities are larger than 50% for most of the game while the home team won. As an example, let's say that the home team maintains a very big lead on the other team (in terms of points) for most of the game, but suddenly loses their lead at the end due to 2 injuries of their star players and thus ultimately losing. Let's also say that the predicted home team win probabilities are very high for most of the game, until the 2 injuries after which they drops a lot. In that case, I would say that the model is fairly accurate. But quantifying its accuracy by looking at which side the graph is most of the time, would deem the model very inaccurate (as it predicted that the home team would win most of the time).

• This isn't really helpful, but: keep in mind that if you measure enough variables, you will have unique values for each game and your graph will tend to either constant 1 or constant 0. So, there has to be a balance re how many factors you use.
– user1566
May 1, 2016 at 19:54
• How do you measure that the 4 features are indeed relevant ? May 2, 2016 at 17:54
• @Romain, well that is exactly the problem. Intuitively, I think the 4 extra features are relevant as they add valuable information. But of course, I don't know how to measure that. May 3, 2016 at 9:54

Interesting problem! If I understand correctly you need to assess the quality of both models and take the one that works best: with better predictive power.

1. Unless you have more games, the results will not be very generalizable. I assume that you have G games.

2. You need to define a metric to compare the 2 models. When looking at binary outcome entropy (or log-loss) is usually an excellent metric. Then you can modify it to fit your purpose. The metric to score a model is always very subjective and especially in your case, you can define many. Here are a few examples:

• Average Entropy (or log-loss) of the probability in the last 10 minutes. In that case, you are measuring how well a model predict "just" before the true answer is known: $$Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{10}\sum_{i=40-10}^{i=40}{[y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}}$$

• Average Entropy (or log-loss) of the probability with exponential loss. In that case, you are measuring how well a model predict "just" before the true answer is known: $$Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{\sum_{i=1}^{i=M}{e^{i-M}}}\sum_{i=1}^{i=M}{e^{i-M} [y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}}$$

With $y_g$ being the outcome of the game (0/1). If you think that the beginning or the middle of the game is more important than you can adjust those function as required.

Hope this help.

• Thanks for your answer! I will try to implement this in a few days and let you know of the results. May 3, 2016 at 9:55
• how did this work for you? @gieldops Mar 12, 2019 at 10:01