I've written a simple model to predict the outcome of soccer games based on some input. The outcome is either "H","D",or "A", short for home team, away team, or draw. I'm comparing the results of this model to predictions made by a popular betting site (meaning both are being compared to the actual outcomes as well).
My question is: what's the best way to check wether my model is performing better/worse than the betting site, and to quantify the amount?
I figured a better way is to compare each model to the actual outcome and replace the outcomes with binary ones just saying true or false based on wether the prediction was correct and then compare this binomial data. I've seen this and this question, that say a paired t-test isn't the ideal choice, suggesting either McNemar's test or a regular z-test. However, I'm not sure it translates to my situation as well, because I'm not familiar enough with the intricacies of each method.
I'm doing all of this in R, so here's some reproducible code for the data:
myTable <- data.frame(outcome=c("H","H","D","A","H"), betsite=c("H","A","A","H","H"), predmodel=c("A","H","D","A","H"), betsitebin=c(TRUE, FALSE, FALSE, FALSE, TRUE), predmodelbin=c(FALSE, TRUE, TRUE, TRUE, TRUE))
Which looks like this:
outcome betsite predmodel betsitebin predmodelbin 1 H H A TRUE FALSE 2 H A H FALSE TRUE 3 D A D FALSE TRUE 4 A H A FALSE TRUE 5 H H H TRUE TRUE