I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the probability that the home team is going to win, throughout the game. I have implemented various machine learning algorithms and use various features (and combinations thereof).
Below is an image showing the win probability of the home team for two settings. Setting 1 uses two simple features while setting 2 uses the same two features plus four other features. The model I used in this case is Naive Bayes (with isotonic calibration). The four extra features seem relevant but ...
I don't know how to know/quantify whether using those four extra features has any impact on the quality of the predictions. This also applies for the other machine learning paradigms I used.
It is impossible to say that a series of predicted win probabilities for a game is accurate because the win probabilities are larger than 50% for most of the game while the home team won. As an example, let's say that the home team maintains a very big lead on the other team (in terms of points) for most of the game, but suddenly loses their lead at the end due to 2 injuries of their star players and thus ultimately losing. Let's also say that the predicted home team win probabilities are very high for most of the game, until the 2 injuries after which they drops a lot. In that case, I would say that the model is fairly accurate. But quantifying its accuracy by looking at which side the graph is most of the time, would deem the model very inaccurate (as it predicted that the home team would win most of the time).
I'm struggling to find any information about this problem on the internet. Does anybody have an idea about evaluating such a series of win probabilities or have some paper about this subject I could read?