How to model and test a decision support system (e.g. a terrorist warning system)? I have reformulated the problem from a "dog barking warning system" to something else which hopefully, has less ambiguity. Instead, I will repose the problem as follows:
Let's assume that my neighbour is a "mad scientist", who claims to have invented a "terrorist event" forecasting machine. The machine has three colored bulbs - one of which will be illuminated, depending on the severity of a terrorist event forecasted by the machine.
The light bulbs have the following interpretation:


*

*No light bulbs illuminated means there is no perceived threat.

*Green bulb illuminated means there is a level 1 threat imminent.

*Orange bulb illuminated there is a level 2 threat imminent.

*Red bulb illuminated there is a level 3 threat imminent.


To avoid getting too pedantic, let's assume for the sake of argument that the following terms are defined and agreed upon:


*

*Level 1 terrorist event.

*Level 2 terrorist event.

*Level 3 terrorist event.

*"imminent" terrorist event.


What I am trying to find out, if there is a way I can design an experiment that can help me say with a degree of confidence, whether the scientists claims are statistically significant or not.
If the claims are found to be statistically significant, then I would like to be able to add a CI (confidence interval) to the claim. So, I can say something like - if the orange bulb is illuminated, then a level 2 terrorist event will occur within a x% CI.
Having said that - IIRC, CI are only meaningful for N~ RV.?
As an aside, I was thinking that something akin to Fischer's tea experiment (or running a Bernoulli test would be useful, but my stats 'fu is not what it used to be).
The purpose of such a model (assuming that the scientist machine really does work), is to act as a decision support system - i.e. decisions can be taken on the output of the machine - if it can be depended upon.
 A: I have thought about this question a while and have come to the following rather vague answer to a, in my eyes, rather vague question. Despite the asker's wish I don't use the word model as I didn't get it into my thinking on this problem. Sorry for that.


*

*What is your hypothesis?
I see three different possible interpretations of the hypothesis:
(a) The one implied by the question: If the dog barks one time, then it rains. (...) This hypotheses would be false if and only if the dog barks one time and it is currently not raining (i.e., a truth functional material implication)
(b) The one that seems more plausible to me: If it rains, then the dog barks exactly one time. (...) This hypotheses would be false if and only if it is raining and the dog does not bark exactly one time
(c) The stochastic hypotheses: Out of a population of dog's the neighbor's dog is a pretty reliable weather indicator compared to the other dogs and barks according to the mentioned scheme.

*How to test your hypotheses?
(a) & (b) Verifying these general claims is pretty difficult. It is like asking if it is really true that the sun always rises in the morning. There are pretty good arguments from philosophers of science as Karl Popper and the like that we can never proof claims like these but only try to falsify them and treat them as true as long as they haven't been falsified. In our example one could do two things: Firstly we could simply survey the weather conditions and the dog's behavior until we get bored and accept the hypotheses as true or see any pairing that is inconsistent with our hypotheses. Secondly, create an environment in which you can control the weather conditions (i.e., run an experiment) take the dog into this environment and proceed with surveying until boredom/falsification. Note however that in case of any falsification one could always argue that e.g., this rain was not really a rain but rather a light drizzle and so on. See Lakatos and also further the Duhem-Quine Thesis. To come back to my example: at some point humanity accepted that the sun always rises in the morning although we can never be sure of it!
(c) To test this claim I would do the following: I would take the neighbor's dog and some random dog from the same population and bring them, separately, to the above mentioned environment where you can control the weather. There you have four different stimuli (S1 = no rain, S2 = rain, ...) which I would present to each dog in a random order and record the reaction to each stimuli. Note that you need to repeat each stimulus several times (e.g., 30 times). Then you could enter the data with stimuli as the subject, the dog as the between-subjects manipulations and the stimuli as the within-subjects manipulations into an Analysis of Variance (ANOVA). Your hypotheses would probably predict an interaction of dog with stimuli. If this would occur you could run tests testing for: the random dog which should have equal barking for all four stimuli, and the neighbor's dog which should stick to the norm (e.g., one sample t-tests against a value). 
Note that your test of hypotheses (c) would be easier and more 'traditional' if you would suspect that there exist two populations of dogs, one weather forecasting and one weather blind population of dogs. Then you could sample some (e.g., 30) from each population and aggregate the data from each stimuli. Dogs would be the subjects, population the between-subject manipulation and stimuli the within-subject manipulation.
Two caveats: I don't know if the described way is the usual one for testing for single outliers . I don't know if there exist more common methods to test for deviance from a norm.
A: I am going to give a simple answer in response to your edit. If my answer does not meet your real needs please include it in the comments and I will change/complexify it a bit.
Let:
$S$ be the severity of the event of interest with higher numbers representing more severity
$W$ be the warning generated by the system.
The range of values that these variables can take is assumed to be from 1 to $n$.
Either via an experiment or via an observational study you have a set of observations about actual events and the warnings generated by the system. Thus, you can calculate the following empirical probabilities:
$P(W=w | S=s)$ which is the probability that the system generates warning $w$ when the event is $s$.
From a decision making perspective, you really want to know:
$P(S=s | W=w)$
In other words, you want to know the probability of different events occurring in the near future given that the system has given you a specific warning. 
You can compute the required probabilities using Bayes theorem. Briefly, the calculation is:
$P(S=s | W=w) = \frac{P(W=w | S=s) P(S=s)}{\sum_s(P(W=w |S=s)P(S=s))}$
From your study you know:
$P(W=w | S=s)$ = Proportion of time the system generated warning $w$ when the event was $s$,
In order to compute the required probability, $P(S=s | W=w)$ you need to have some sense of $P(S=s)$ which you can estimate or guess based on previous experience.
You can then assess the accuracy of the device by looking at the following probabilities: $P(S=s|W=s)$.
A: Count sucesses an failures when the barking patterns occur and get a contingency table. Then do a Chi squared test against 50/50 occurences (ie null hypothesis is that barking is not correlated with getting the correct weather pattern).
Of course, there is also the other side of this to consider: what barking pattern precedes a particular weather type.
