Evaluating the rate of convergence for MLE of multinomial distribution

Question

I'd like quantify how the number of trials $n$ influences the quality of an MLE estimate of multinomial distribution with even probabilities $p_1,...,p_k$. So we know that we can estimate the event probabilities as $\theta_i = \frac{n_i}{n}$ and for I noticed that for distribution relying on a single parameter, like the Poisson distribution, people tend to simply calculate the Variance of this one parameter, or in the case of the normal distribution, one calculates the variance of the sample mean and sample variance independently. Now for the multinomial distribution, it would be nice to arrive at some statement how well the estimated probability vector $\theta$ matches the ground truth probability vector $p$, so I guess I'd like to estimate the following quantity as a function of $n$: $$ E[ d(\theta,p)], $$ where $d$ is the Euclidean distance. I guess it would be easier to compute the variance of the individual entries in $\theta_i$, i.e. $E[(\frac{n_i}{n} - p_i)^2]$, but is it even possible to estimate the former quantity? Or is there another 'more standard' way to quantify the convergence of such an estimator?

Thank's a lot for your help, Max

Answer 1

I don't think the following upper bound is tight, but it's an upper bound nevertheless. First notice that $d(\theta,p) = \sqrt{||\theta - p||_2^2}$, and since the square root is concave, using Jensen's inequality and the linearity of the expectation we get

$$ E[d(\theta,p)] \leq \sqrt{E[||\theta - p||_2^2]} = \sqrt{\sum_{i=1}^kE\left[(\theta_i - p_i\right)^2] }\quad. $$ We did this because, as you've said, it's easier to work with the variances. Note that the estimator can be written as $\theta_i = \frac{1}{n}\sum_{j=1}^n 1_{\{X_j =i\}}$, hence $Var(\theta_i) = \frac{p_i(1-p_i)}{n}.$

Before giving the bound, notice that $\forall p\,\in\,[0,1],\,p(1-p)\leq\frac{1}{4}$, so $Var(\theta_i) \leq \frac{1}{4n}$. Finally

$$ E[d(\theta,p)] \leq \sqrt{\sum_{i=1}^kE\left[(\theta_i - p_i\right)^2] } = \sqrt{\sum_{i=1}^kVar\left(\theta_i\right) } \leq \sqrt{\sum_{i=1}^k \frac{1}{4n}}\\ \implies E[d(\theta,p)] \leq \frac{1}{2}\sqrt{\frac{k}{n}} $$