Comparing and evaluating win probabilities in sports from different settings

Question

Background

I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the probability that the home team is going to win, throughout the game. I have implemented various machine learning algorithms and use various features (and combinations thereof).

Below is an image showing the win probability of the home team for two settings. Setting 1 uses two simple features while setting 2 uses the same two features plus four other features. The model I used in this case is Naive Bayes (with isotonic calibration). The four extra features seem relevant but ...

The problem

I don't know how to know/quantify whether using those four extra features has any impact on the quality of the predictions. This also applies for the other machine learning paradigms I used.

It is impossible to say that a series of predicted win probabilities for a game is accurate because the win probabilities are larger than 50% for most of the game while the home team won. As an example, let's say that the home team maintains a very big lead on the other team (in terms of points) for most of the game, but suddenly loses their lead at the end due to 2 injuries of their star players and thus ultimately losing. Let's also say that the predicted home team win probabilities are very high for most of the game, until the 2 injuries after which they drops a lot. In that case, I would say that the model is fairly accurate. But quantifying its accuracy by looking at which side the graph is most of the time, would deem the model very inaccurate (as it predicted that the home team would win most of the time).

I'm struggling to find any information about this problem on the internet. Does anybody have an idea about evaluating such a series of win probabilities or have some paper about this subject I could read?

Answer 1

Interesting problem! If I understand correctly you need to assess the quality of both models and take the one that works best: with better predictive power.

1. Unless you have more games, the results will not be very generalizable. I assume that you have G games.

2. You need to define a metric to compare the 2 models. When looking at binary outcome entropy (or log-loss) is usually an excellent metric. Then you can modify it to fit your purpose. The metric to score a model is always very subjective and especially in your case, you can define many. Here are a few examples:

   • Average Entropy (or log-loss) of the probability in the last 10 minutes. In that case, you are measuring how well a model predict "just" before the true answer is known: $$ Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{10}\sum_{i=40-10}^{i=40}{[y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}} $$

   • Average Entropy (or log-loss) of the probability with exponential loss. In that case, you are measuring how well a model predict "just" before the true answer is known: $$ Error = \frac{1}{G}\sum_{g=1}^{g=G}{\frac{1}{\sum_{i=1}^{i=M}{e^{i-M}}}\sum_{i=1}^{i=M}{e^{i-M} [y_g*log(p_{i}^{g})+(1-y_g)*log((1-p_{i}^{g}))]}} $$

With $y_g$ being the outcome of the game (0/1). If you think that the beginning or the middle of the game is more important than you can adjust those function as required.

Hope this help.