4
$\begingroup$

There are four models I'm looking at and trying to evaluate which will give me the best predictions going forward. So I loop through each model and look at the error rate, the Brier Error and the AIC. In some cases the AIC and Brier Error are not consistent (results for model 3 and 4 below). If they give contrasting indications, which one should be paid attention to? Why? Or am I looking at this the wrong way? For those interested, my R code and results are below:

line<- seq(71,79, by=.5)

models<- list("target~total+tot_eit_h_h1+tot_both_h_h1","target~total", "target~total*spread*as.factor(loc)", "target~total+spread*as.factor(loc)")

    for(j in 1:length(models)){
       for(i in 1:length(line)){

    data$target <- ifelse(data$result>line[i], 1, 
                          ifelse(data$result<line[i], 0, 
                                 rbinom(dim(data)[1],1,.5)))

    model<-glm(models[[j]], data=data, family="binomial")
    data$prediction <- predict(model, data, type='response') 

    brier=with(data, mean((prediction-target)^2))
    error_rate <- mean((data$prediction>0.5 & data$target==0) | (data$prediction<0.5 & data$target==1))
    aics<-c(aics, model$aic)
    briers<-c(briers, brier)
    error_rates<-c(error_rates, error_rate)
    }
  print(paste(mean(error_rates), "error rates"))
  print(paste(mean(aics), "AIC"))
  print(paste(mean(briers), "BRIER"))
  cat("\n")
  error_rates<-c()
  aics<-c()
  briers<-c()

}

RESULTS

[1] "0.381418853255588 error rates"
[1] "1633.50190616553 AIC"
[1] "0.232600213678421 BRIER"

[1] "0.38491739552964 error rates"
[1] "1641.54699258567 AIC"
[1] "0.234847243715002 BRIER"

[1] "0.314091350826045 error rates"
[1] "1451.51883987259 AIC"
[1] "0.200957523053587 BRIER"

[1] "0.324509232264334 error rates"
[1] "1467.88024134523 AIC"
[1] "0.205168360087164 BRIER"
$\endgroup$
5
  • $\begingroup$ Did you have enough samples that AICc was not required? $\endgroup$ Aug 12, 2013 at 2:31
  • $\begingroup$ EngrStudent, I'm not sure what you mean by your question but I have 1200 data points, and they have very big variance, so it seems doubtful there are sufficient samples... $\endgroup$
    – appleLover
    Aug 12, 2013 at 2:35
  • 3
    $\begingroup$ If by "inconsistent" you mean that the model with the lowest Brier score isn't the one with the lowest AIC, the answer is there's no reason to expect them to be. AIC's based on a likelihood measure of fit, & more importantly it penalizes bigger models to avoid overfitting - that's the point of it. $\endgroup$ Aug 12, 2013 at 9:19
  • 1
    $\begingroup$ +1. More minutely, Brier was G.W. Brier, so "Brier score" is fine, "Brier's score" is grammatical, if less commonly seen in my experience, but "Briers" is a typo. $\endgroup$
    – Nick Cox
    Aug 12, 2013 at 9:25
  • $\begingroup$ I made the edits that Frank suggests below. Interestingly, the inconsistency cleared up and now by all measures the third model is best. However, when I use these results and the third model testing this data in a real world situation it performs poorly. Model two, which appears to be the worst, does by far the best in a real world situation. Pretty disappointed by these statistical tests. $\endgroup$
    – appleLover
    Aug 12, 2013 at 21:01

1 Answer 1

4
$\begingroup$

You need to divide by $n$ to get the Brier score, and there should be no 100 in its formula. It is just the mean squared error. The "error rate" is not meaningful and will give misleading results, so I would ignore it. AIC is a measure of predictive discrimination whereas the Brier score is a combined measure of discrimination + calibration.

models can be a list of models, e.g.

models <- list(y ~ x1, y ~ x1 + x2, ...)

What does line represent and should you really be fitting ordinal regression models?

Please update your code to show us the correct Brier score, along with AIC.

$\endgroup$
4
  • $\begingroup$ I updated the code and results above with your comments. To explain what my code does; this is used to predict scores in the first half of women's basketball games. I predict over multiple lines because for any given game, the line may be some number between 70 and 80. I use logistic regression because for any given game, I will only need to decide if it is over or under one specific number. Maybe linear regression with some cumulative distribution function would be worth trying, that idea occurred to me but in the end, I need a prediction on a binary outcome so maybe not a good idea. $\endgroup$
    – appleLover
    Aug 12, 2013 at 21:10
  • $\begingroup$ Based on the results above, the third model seems by far the best. Also based on my domain knowledge, it seems reasonable. But when I test it using real betting lines it gets destroyed. The simple model two (also reasonable based on my domain knowledge) tests the best by far when using real betting lines. $\endgroup$
    – appleLover
    Aug 12, 2013 at 21:13
  • $\begingroup$ You still do not have the correct formula for Brier score. Use with(data, mean((predicted-target)^2)). $\endgroup$ Aug 12, 2013 at 21:24
  • $\begingroup$ fixed and updated $\endgroup$
    – appleLover
    Aug 12, 2013 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.