My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises naturally, and is in fact the definition of dimension, in quantum mechanical models.)

I'll give a simple analogy. Suppose I have a $d$-sided die I wish to know the bias of. But, moreover, I do not know $d$. If I've have seen $k$ distinct outcomes, then I know there is at least $k$ sides. But, perhaps there is more?

My initial thought was to use model selection criteria, such as AIC. I could, of course, use AIC, but first I would have assume there is some maximum value of $k$. But this won't be practically relevant since the maximum likelihood term in the AIC will be the same regardless of how many additional sides I add to the die because the maximum likelihood estimator of the probability of those sides will be 0. Then, the model with the best AIC will be the one with the number of distinct observed sides: $k$. Seems overconfident to me. Moreover, I'd like a technique that doesn't require an assumption on the maximum number of possible outcomes.

My question, then, is: are there model selection techniques other than AIC which can hedge between the absolute minimal description and ones that favor models which predict outcomes not yet observed?

  • $\begingroup$ I'm guessing that something Bayesian would be appropriate. Without any data on the hypothetical unobserved sides, you might have to inject some kind of prior belief. Can you clarify what are at least two of your possible candidate models? $\endgroup$ – zkurtz Sep 16 '13 at 20:26
  • $\begingroup$ Well, my exact problem would add too much detail. But, a simple proxy is the multinomial (dice) model I mentioned. That is, the family of models is equivalent to a set of dice with every possible number of sides. In this case, you are suggesting placing a prior on the number of sides? I imagine something that decreases exponentially on the number of sides? $\endgroup$ – Chris Ferrie Sep 17 '13 at 15:25

Your problem is quite similar to estimating an upper bound of a uniform distribution, only your distribution is discrete. It is true that MLE estimate is equal to the maximal observed value, but, as explained here, the minimal variance unbiased estimator is greater than the maximal value. They also give an example of Bayesian analysis with a flat improper prior. Again, the expected value of the posterior turns out to be greater than the maximal observation.

After you obtain the posterior, you can compute the likelihood and AIC for any given number of dice facets using a simple conditional probability (density) formula.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.