9
$\begingroup$

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each t.

The plot of P(Event) versus time is as given in the figure below. In this figure, the black line represents P(Event) as predicted by my model; the horizontal red line represents the prior probability of the event happening; and the dotted vertical lines represent the (five) event occurrences on the time series.

Ideally, I wish to see the predicted P(Event) peak prior to observing any events and remain close to zero when there is no prospect of an event.

P(Event) versus Time Graph

I want to be able to report how well my model (the black line) performs in predicting the event occurrences. An obvious candidate to compare my model with is the prior probability of event (the red line), which -if used as a predictor- would predict the same probability value for all t.

What is the best formal method to achieve this comparison?

P.S: I am currently using the (intuitive) scoring as coded below, where an overall lower score indicates better prediction performance. I found that it is actually quite difficult to beat the prior with this scoring:

# Get prediction performance
model_score = 0; prior_score=0; 

for t in range(len(timeSeries)):

   if(timeSeries[t]== event):  # event has happened
      cur_model_score = 1- prob_prediction[t]; 
      cur_prior_score = 1 - prior
   else: # no event
      cur_model_score = prob_prediction[t] - 0;
      cur_prior_score = prior - 0;

   model_score = model_score + abs(cur_model_score);
   prior_score = prior_score + abs(cur_prior_score);
$\endgroup$
2
  • $\begingroup$ Do you think you could impose a meaningful loss function? Is there a way to say how much you gain/lose by guessing correctly/incorrectly? $\endgroup$
    – James
    Aug 17, 2014 at 19:30
  • $\begingroup$ I thought of this, but I do not want to invent an arbitrary loss function that I can tweak to achieve low or high performance as I wish. $\endgroup$
    – Zhubarb
    Aug 18, 2014 at 9:38

1 Answer 1

1
$\begingroup$

You can create a ROC curve. For a given value of p between 0 and 1 you predict that the event is going to happen if the predicted probability is greater than p. Then you calculate TPR and FPR which gives you a single point on the ROC curve. By varying p between zero and one you obtain the entire curve. E.g. for p < 0.005 the prior-based predictor will always say that the event will happen at all times.

For more, see:

http://en.wikipedia.org/wiki/Receiver_operating_characteristic

$\endgroup$
1
  • $\begingroup$ Just to add that AUC is available as a summary statistic for these curves. $\endgroup$ Nov 28, 2014 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.