# Model selection with different outcome spaces

My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises naturally, and is in fact the definition of dimension, in quantum mechanical models.)

I'll give a simple analogy. Suppose I have a $d$-sided die I wish to know the bias of. But, moreover, I do not know $d$. If I've have seen $k$ distinct outcomes, then I know there is at least $k$ sides. But, perhaps there is more?

My initial thought was to use model selection criteria, such as AIC. I could, of course, use AIC, but first I would have assume there is some maximum value of $k$. But this won't be practically relevant since the maximum likelihood term in the AIC will be the same regardless of how many additional sides I add to the die because the maximum likelihood estimator of the probability of those sides will be 0. Then, the model with the best AIC will be the one with the number of distinct observed sides: $k$. Seems overconfident to me. Moreover, I'd like a technique that doesn't require an assumption on the maximum number of possible outcomes.

My question, then, is: are there model selection techniques other than AIC which can hedge between the absolute minimal description and ones that favor models which predict outcomes not yet observed?

• I'm guessing that something Bayesian would be appropriate. Without any data on the hypothetical unobserved sides, you might have to inject some kind of prior belief. Can you clarify what are at least two of your possible candidate models? – zkurtz Sep 16 '13 at 20:26
• Well, my exact problem would add too much detail. But, a simple proxy is the multinomial (dice) model I mentioned. That is, the family of models is equivalent to a set of dice with every possible number of sides. In this case, you are suggesting placing a prior on the number of sides? I imagine something that decreases exponentially on the number of sides? – Chris Ferrie Sep 17 '13 at 15:25