How can I handle discrete events in a continuous time stream in the context of an F1 metric?

To give an example, let's say the Earthquake Forecasting Bureau would report the following for their performance in 2021:

No quake predicted Mild quake predicted Severe quake predicted
No quake occurred X 2 3
Mild quake occurred 4 5 6
Severe quake occurred 7 8 9

What to fill in for the X? The Earthquake Forecasting Bureau argues that their conference room calendar allows to schedule 2400 press conferences per year and therefore X=2400 and their F1 score for 2021 is 99.9%

But now they are applying for funding for another conference room so that next year they can improve their F1 score to 99.95%.

Clearly, this makes no sense. But how can we fix this?

  • $\begingroup$ Could you elaborate on what a "fix" is intended to achieve? The trap you might be about to stumble into is that no matter what value you enter for "X," your table cannot legitimately be analyzed using standard techniques that assume the observations (not the table columns!) are independent. To analyze data like these you need to apply methods that model the temporal correlation: that is, time series or stochastic process methods. Trying to complete this table with a value of "X" is likely to be a dead end. $\endgroup$
    – whuber
    Aug 18 at 12:08
  • $\begingroup$ Thank you for your answer. One fix would be to force X=0 (probably the only defensible number for X). But this general problem doesn't seem too far fetched and I was surprised I couldn't find any discussion of it $\endgroup$
    – KevinB56
    Aug 20 at 14:48
  • $\begingroup$ A general term for this is marked point process. Because constructing a table of this sort is not generally an effective way to analyze such phenomena, we shouldn't be surprised (or even concerned) at not finding discussions of it. $\endgroup$
    – whuber
    Aug 20 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.