12
$\begingroup$

If there are multiple possible approximations, I'm looking for the most basic one.

$\endgroup$
22
$\begingroup$

You can approximate it with the multivariate normal distribution in the same way that binomial distribution is approximated by univariate normal distribution. Check Elements of Distribution Theory and Multinomial Distribution pages 15-16-17.

Let $P=(p_1,...,p_k)$ be the vector of your probabilities. Then the mean vector of the multivariate normal distribution is $ np=(np_1,np_2,...,np_k)$. The covariance matrix is a $k \times k$ symmetric matrix. The diagonal elements are actually the variance of $X_i$'s; i.e.$ np_i(1-p_i)$, $i=1,2...,k$. The off-diagonal element in the ith row and jth column is $\text{Cov}(X_i,X_j)=-np_ip_j$, where $i$ is not equal to $j$.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Check out the 2nd reference. $\endgroup$ – Stat Aug 17 '12 at 21:06
  • 3
    $\begingroup$ Stat, so that this answer can stand by itself (and be resistant to link rot), would you mind giving a summary of the solution? $\endgroup$ – whuber Aug 17 '12 at 21:32
  • 4
    $\begingroup$ Does this need a continuity correction? How would you apply it? $\endgroup$ – Jack Aidley May 23 '14 at 11:04
  • 2
    $\begingroup$ The covariance matrix is not positive definite, but rather positive semi-definite, and is not full-rank. This makes the resulting multinormal distribution undefined. This is the problem I faced. Any idea how to handle it? $\endgroup$ – Mohammad Alaggan Sep 13 '16 at 12:29
  • 2
    $\begingroup$ @M.Alaggan: The mean/covariance matrices defined here have one minor issue: For a multinomial distribution with $k$ variables, the equivalent multivariate normal has $k-1$ variates. This is evident in the simple binomial example, which is approximate by the (ordinary) normal distribution. For further discussion, see Example 12.7 of Elements of Distribution Theory. $\endgroup$ – M.S. Dousti Jul 21 '17 at 17:34
1
$\begingroup$

The density given in this answer is degenerate, and so I used the following to calculate the density that results from the normal approximation:

There's a theorem that says given a random variable $X = [X_1, \ldots, X_m]^T \sim \text{Multinom}(n, p)$, for an $m$-dimensional vector $p$ with $\sum_i p_i = 1$ and $\sum_i X_i = n$, that;

$$ X \xrightarrow{d} \sqrt{n} \, \text{diag}(u) \, Q \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \\ 0 \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_m \end{bmatrix}, $$

for large $n$, given;

  • a vector $u$ with $u_i = \sqrt{p_i}$;
  • random variables $Z_i \sim N(0,1)$ for $i = 1, \ldots, m-1$, and;
  • an orthogonal matrix $Q$ with final column $u$.

That is to say, with some rearrangement, we can work out an $m-1$ dimensional multivariate normal distribution for the first $m-1$ components of $X$ (which are the only interesting components because $X_m$ is the sum of the others).

A suitable value of the matrix $Q$ is $I - 2 v v^T$ with $v_i = (\delta_{im} - u_i) / \sqrt{2(1 - u_m)}$ - i.e. a particular Householder transformation.

If we restrict the left-hand side to the first $m-1$ rows, and restrict $Q$ to its first $m-1$ rows and $m-1$ columns (denote these $\hat{X}$ and $\hat{Q}$ respectively) then:

$$ \hat{X} \xrightarrow{d} \sqrt{n} \text{diag}(\hat{u}) \hat{Q} \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_{m-1} \end{bmatrix} \sim \mathcal{N} \left( \mu, n \Sigma \right), $$

for large $n$, where;

  • $\hat{u}$ denotes the first $m-1$ terms of $u$;
  • the mean is $\mu = [ n p_1, \ldots, n p_{m-1}]^T$, and;
  • the covariance matrix $n \Sigma = n A A^T$ with $A = \text{diag}( \hat{u} ) \hat{Q}$.

The right hand side of that final equation is the non-degenerate density used in calculation.

As expected, when you plug everything in, you get the following covariance matrix:

$$ (n\Sigma)_{ij} = n \sqrt{p_i p_j} (\delta_{ij} - \sqrt{p_i p_j}) $$

for $i,j = 1, \ldots, m-1$, which is exactly the covariance matrix in the original answer restricted to its first $m-1$ rows and $m-1$ columns.

This blog entry was my starting point.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Another useful resource is the links provided in: stats.stackexchange.com/questions/2397/… $\endgroup$ – stephematician Mar 17 at 5:21
  • 1
    $\begingroup$ Good answer (+1) --- Note that you can embed links with the syntax [textual description](hyperlink). I have taken the liberty of editing this answer to embed your links. $\endgroup$ – Ben - Reinstate Monica Mar 17 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.