I'd like quantify how the number of trials $n$ influences the quality of an MLE estimate of multinomial distribution with even probabilities $p_1,...,p_k$. So we know that we can estimate the event probabilities as $\theta_i = \frac{n_i}{n}$ and for I noticed that for distribution relying on a single parameter, like the Poisson distribution, people tend to simply calculate the Variance of this one parameter, or in the case of the normal distribution, one calculates the variance of the sample mean and sample variance independently. Now for the multinomial distribution, it would be nice to arrive at some statement how well the estimated probability vector $\theta$ matches the ground truth probability vector $p$, so I guess I'd like to estimate the following quantity as a function of $n$: $$ E[ d(\theta,p)], $$ where $d$ is the Euclidean distance. I guess it would be easier to compute the variance of the individual entries in $\theta_i$, i.e. $E[(\frac{n_i}{n} - p_i)^2]$, but is it even possible to estimate the former quantity? Or is there another 'more standard' way to quantify the convergence of such an estimator?

Thank's a lot for your help, Max


1 Answer 1


I don't think the following upper bound is tight, but it's an upper bound nevertheless. First notice that $d(\theta,p) = \sqrt{||\theta - p||_2^2}$, and since the square root is concave, using Jensen's inequality and the linearity of the expectation we get

$$ E[d(\theta,p)] \leq \sqrt{E[||\theta - p||_2^2]} = \sqrt{\sum_{i=1}^kE\left[(\theta_i - p_i\right)^2] }\quad. $$ We did this because, as you've said, it's easier to work with the variances. Note that the estimator can be written as $\theta_i = \frac{1}{n}\sum_{j=1}^n 1_{\{X_j =i\}}$, hence $Var(\theta_i) = \frac{p_i(1-p_i)}{n}.$

Before giving the bound, notice that $\forall p\,\in\,[0,1],\,p(1-p)\leq\frac{1}{4}$, so $Var(\theta_i) \leq \frac{1}{4n}$. Finally

$$ E[d(\theta,p)] \leq \sqrt{\sum_{i=1}^kE\left[(\theta_i - p_i\right)^2] } = \sqrt{\sum_{i=1}^kVar\left(\theta_i\right) } \leq \sqrt{\sum_{i=1}^k \frac{1}{4n}}\\ \implies E[d(\theta,p)] \leq \frac{1}{2}\sqrt{\frac{k}{n}} $$

  • $\begingroup$ Thank you so much, this was incredibly useful. $\endgroup$
    – Maximal
    Commented Apr 9, 2019 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.