What are some basic principles to handle "none of the above" events in Bayesian statistics?

Question

Suppose you want to assign a noninformative prior to the following event:

The next tree that we will encounter is a:

1. Spruce
2. Pine
3. None of the above

We don't have any prior information, so we are tempted to give each choice the same probability value, i.e. 1/3. But is this the right approach? Are there some general principles that scientists use to deal with "catch all else" categories?

Furthermore, suppose that we learn that a tree named Sequoia exists:

1. Spruce
2. Pine
3. Sequoia
4. None of the above

Now the question has four possible answers and the prior probability of "None of the above" has shrunk to 1/4. Same as the probability of any other tree. This somehow feels disturbing, because there is some inherent special status that "None of the above" should have. A Sequoia feels to have an affinity to a Spruce, but not to an abstract concept of "Any other tree".

Surely I am not the first person disturbed by this, so I would like to know if there are some general guidelines that one should follow when one category is defined negatively by all other categories.

Answer 1

This sounds a bit like a scenario that calls for Dirichlet or Chinese Restaurant processes.

The Dirichlet distribution is commonly used as a prior for multinomial or categorical distributions (to which it is prior), but these require knowing the number of possible categories in advance. To solve this, people sometimes turn to processes, which essentially probability distributions over probability distributions.

The Chinese Restaurant Process is usually described with this analogy. Suppose there was a giant Chinese restaurant with countably-infinite number of round tables*, numbered starting at 1. The first customer to enter the restaurant sits down at Table 1. The subsequent customers either "found" a new table, with probability $\frac{\alpha}{n - 1 + \alpha}$ or join the customers already at the $k$th table with probability $\frac{c_k}{n - 1 + \alpha}$, where $c_k$ is the number of customers at the $k$th table and $n$ customers total.

This seems very similar to your scenario, where one is likely to see more of the trees already seen the most, but might encounter a rare new species too.

I think the original reference is Aldous (1985). I liked this tutorial by Gershman and Blei (2011), and it has become a pretty standard topic in Bayesian non-parametric methods.

---

* The tables are invariably described as 'round', and the restaurant 'Chinese'. I have no idea why the roundness is emphasized, but the Chinese part is allegedly it is due to the seemingly-infinite seating capacity of Chinese restaurants in San Francisco. If your group memberships are not 'hard' and a data point can partially belong to multiple groups, you may want to look into the amusingly-named Indian Buffet process instead.

Answer 2

This is a classic Principle of Insufficient Reason question. The problem arises because you do have prior information.

If you had instead asked: "The next tree that we will encounter is a 1. Flob 2. Blorb 3. Glob" (where the meaning of Flob, Blorb and Glob is unknown to you) then it would be reasonable, according to the principle, to assign equal probabilities.

But in fact, you do know that one of your options is "None of the above". The reason why the uniform prior makes you uncomfortable is because you feel that there should be more of the "none of the above" type of tree than of spruce or pine. This feeling is prior information which you should be including in your prior.

For the second question, this is a criticism of the Principle of Insufficient Reason which has been articulated by many authors. However, I don't think it really matters here, as long as you give the same probability to each of spruce, pine and sequoia if you think there is nothing to choose between them.

For the final question, the general guideline would be: imagine a forest, say of 100 trees. Count the number of pine, spruce and sequoia trees, say 10 of each. Then your prior probability of "none of the above" should be 0.7. If you find these numbers absurd, well, that's because you know more about forests than I do! You should be using that knowledge to choose your prior. Reasoning from a position of pretend-ignorance definitely has its place in statistics, but not here.