My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises naturally, and is in fact the definition of dimension, in quantum mechanical models.)
I'll give a simple analogy. Suppose I have a $d$-sided die I wish to know the bias of. But, moreover, I do not know $d$. If I've have seen $k$ distinct outcomes, then I know there is at least $k$ sides. But, perhaps there is more?
My initial thought was to use model selection criteria, such as AIC. I could, of course, use AIC, but first I would have assume there is some maximum value of $k$. But this won't be practically relevant since the maximum likelihood term in the AIC will be the same regardless of how many additional sides I add to the die because the maximum likelihood estimator of the probability of those sides will be 0. Then, the model with the best AIC will be the one with the number of distinct observed sides: $k$. Seems overconfident to me. Moreover, I'd like a technique that doesn't require an assumption on the maximum number of possible outcomes.
My question, then, is: are there model selection techniques other than AIC which can hedge between the absolute minimal description and ones that favor models which predict outcomes not yet observed?