I have reformulated the problem from a "dog barking warning system" to something else which hopefully, has less ambiguity. Instead, I will repose the problem as follows:
Let's assume that my neighbour is a "mad scientist", who claims to have invented a "terrorist event" forecasting machine. The machine has three colored bulbs - one of which will be illuminated, depending on the severity of a terrorist event forecasted by the machine.
The light bulbs have the following interpretation:
- No light bulbs illuminated means there is no perceived threat.
- Green bulb illuminated means there is a level 1 threat imminent.
- Orange bulb illuminated there is a level 2 threat imminent.
- Red bulb illuminated there is a level 3 threat imminent.
To avoid getting too pedantic, let's assume for the sake of argument that the following terms are defined and agreed upon:
- Level 1 terrorist event.
- Level 2 terrorist event.
- Level 3 terrorist event.
- "imminent" terrorist event.
What I am trying to find out, if there is a way I can design an experiment that can help me say with a degree of confidence, whether the scientists claims are statistically significant or not.
If the claims are found to be statistically significant, then I would like to be able to add a CI (confidence interval) to the claim. So, I can say something like - if the orange bulb is illuminated, then a level 2 terrorist event will occur within a x% CI.
Having said that - IIRC, CI are only meaningful for N~ RV.?
As an aside, I was thinking that something akin to Fischer's tea experiment (or running a Bernoulli test would be useful, but my stats 'fu is not what it used to be).
The purpose of such a model (assuming that the scientist machine really does work), is to act as a decision support system - i.e. decisions can be taken on the output of the machine - if it can be depended upon.