# How should I measure my confidence in this estimate of this finite distribution

Suppose I have a six sided die, that may not be fair. The probabilities for the six sides are equally likely to be any values in the interval [0,1], subject to the constraint that they must add up to 1. Suppose I wish to estimate this finite distribution. One plausible option is that after n throws of the die, I should estimate p1 as number of 1s rolled over n, and so on up to p6.

If I were to proceed in this way, how should I measure/quantify my confidence in this estimate of this finite distribution?

If I don't proceed this way, what other approaches are there to estimate the distribution on the basis of the observed evidence? Should I start from the assumption that the die is fair?

• "Clearly after n throws of the die, I should estimate p1 as number of 1s rolled over n, and so on up to p6." Why is that clear? – Mark L. Stone Sep 29 '15 at 15:32
• Your remark "clearly ... I should estimate..." hides a strong unstated assumption: this estimate is good only for certain loss functions. One can conceive of circumstances where this estimate would perform poorly, such as when the die will be used in a game of chance where it is of paramount importance not to underestimate the chance of rolling a six. Awareness of this assumption is important, because your loss function ought to determine how to quantify the goodness of the estimate. – whuber Sep 29 '15 at 15:33
• Agreed, have edited and hopefully resolved this – EBartrum Sep 29 '15 at 15:46

First of all when you say "the probabilities for the six sides are equally likely to be any values in the interval [0,1], subject to the constraint that they must add up to 1," unfortunately this does not make sense. If you are looking for a way of describing a distribution on dice (Multinomial Distributions) you may want to look into the Dirichlet Distribution. Also, if you're interested in thinking about the probability of different dice, as opposed to viewing your die as a fixed entity with fixed parameters, you may want to consider Bayesian Credible Intervals instead.

• All of that being said, if you are focusing on one of the sides (for example you focused your comments on $p_1$) then your die might as well be a coin, that either lands on side 1 or does not. In that sense you can obtain confidence intervals based on the usual confidence intervals of proportions. That being said do not construct multiple confidence intervals for the different faces and expect them to behave independently, because one interval not containing the true value of the parameter will increase the probability that another does not.

• If you would like to consider and confidence "region" as opposed to an interval, there is apparently some study of this that can be found by searching "Multinomial Confidence Interval". This reference gives you some sense of the asymptotic error of the estimates you suggested http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.html.

• Lastly, I suggest you look into the Dirichlet-Multinomial Distribution. The Bayesian approach to probability is focused on modelling our uncertainty about certain parameters, and can be much more general and interpretable than confidence intervals.

Edit

Yes there is precisely such an approach. Let me give you the rough sketch and hopefully enough jargon so that you can continue.

A Dirichlet Distribution of order $k \geq 2$ is a distribution that is only positive on the $(k-1)$-simplex. This simplex is precisely the set of $k$-dimensional vectors with positive coordinates that sum to 1. When you say you want to start off with some uniform assumption, what you want is a Uniform Distribution on this simplex, which happens to be a special case of the Dirichlet Distribution. The Dirichlet Distribution is closely related to the Multinomial Distribution because it is conjugate prior. This means that if you assume your die came from a certain Dirichlet Distribution, and then you observe data (roll your die) and updat your belief (distribution) about where your die came from, you still get a Dirichlet.

So to give you an idea:

• Start with a uniform distribution $Dir(6, (1,1,1,1,1,1))$ on 6-sided die.
• After seeing $n_i$ tosses for each face you update to $Dir(6, (n_1+1, \dots, n_6+1))$. You still have a distribution modelling your uncertainty.