16
$\begingroup$

If there are multiple possible approximations, I'm looking for the most basic one.

$\endgroup$

2 Answers 2

23
$\begingroup$

You can approximate it with the multivariate normal distribution in the same way that binomial distribution is approximated by univariate normal distribution. Check Elements of Distribution Theory and Multinomial Distribution pages 15-16-17.

Let $P=(p_1,...,p_k)$ be the vector of your probabilities. Then the mean vector of the multivariate normal distribution is $ np=(np_1,np_2,...,np_k)$. The covariance matrix is a $k \times k$ symmetric matrix. The diagonal elements are actually the variance of $X_i$'s; i.e.$ np_i(1-p_i)$, $i=1,2...,k$. The off-diagonal element in the ith row and jth column is $\text{Cov}(X_i,X_j)=-np_ip_j$, where $i$ is not equal to $j$.

$\endgroup$
9
  • 1
    $\begingroup$ Check out the 2nd reference. $\endgroup$
    – Stat
    Aug 17, 2012 at 21:06
  • 3
    $\begingroup$ Stat, so that this answer can stand by itself (and be resistant to link rot), would you mind giving a summary of the solution? $\endgroup$
    – whuber
    Aug 17, 2012 at 21:32
  • 4
    $\begingroup$ Does this need a continuity correction? How would you apply it? $\endgroup$ May 23, 2014 at 11:04
  • 2
    $\begingroup$ The covariance matrix is not positive definite, but rather positive semi-definite, and is not full-rank. This makes the resulting multinormal distribution undefined. This is the problem I faced. Any idea how to handle it? $\endgroup$ Sep 13, 2016 at 12:29
  • 2
    $\begingroup$ @M.Alaggan: The mean/covariance matrices defined here have one minor issue: For a multinomial distribution with $k$ variables, the equivalent multivariate normal has $k-1$ variates. This is evident in the simple binomial example, which is approximate by the (ordinary) normal distribution. For further discussion, see Example 12.7 of Elements of Distribution Theory. $\endgroup$ Jul 21, 2017 at 17:34
1
$\begingroup$

The density given in this answer is degenerate, and so I used the following to calculate the density that results from the normal approximation:

There's a theorem that says given a random variable $X = [X_1, \ldots, X_m]^T \sim \text{Multinom}(n, p)$, for an $m$-dimensional vector $p$ with $\sum_i p_i = 1$ and $\sum_i X_i = n$, that;

$$ X \xrightarrow{d} \sqrt{n} \, \text{diag}(u) \, Q \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \\ 0 \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_m \end{bmatrix}, $$

for large $n$, given;

  • a vector $u$ with $u_i = \sqrt{p_i}$;
  • random variables $Z_i \sim N(0,1)$ for $i = 1, \ldots, m-1$, and;
  • an orthogonal matrix $Q$ with final column $u$.

That is to say, with some rearrangement, we can work out an $m-1$ dimensional multivariate normal distribution for the first $m-1$ components of $X$ (which are the only interesting components because $X_m$ is the sum of the others).

A suitable value of the matrix $Q$ is $I - 2 v v^T$ with $v_i = (\delta_{im} - u_i) / \sqrt{2(1 - u_m)}$ - i.e. a particular Householder transformation.

If we restrict the left-hand side to the first $m-1$ rows, and restrict $Q$ to its first $m-1$ rows and $m-1$ columns (denote these $\hat{X}$ and $\hat{Q}$ respectively) then:

$$ \hat{X} \xrightarrow{d} \sqrt{n} \text{diag}(\hat{u}) \hat{Q} \begin{bmatrix} Z_1 \\ \vdots \\ Z_{m-1} \end{bmatrix} + \begin{bmatrix} n p_1 \\ \vdots \\ n p_{m-1} \end{bmatrix} \sim \mathcal{N} \left( \mu, n \Sigma \right), $$

for large $n$, where;

  • $\hat{u}$ denotes the first $m-1$ terms of $u$;
  • the mean is $\mu = [ n p_1, \ldots, n p_{m-1}]^T$, and;
  • the covariance matrix $n \Sigma = n A A^T$ with $A = \text{diag}( \hat{u} ) \hat{Q}$.

The right hand side of that final equation is the non-degenerate density used in calculation.

As expected, when you plug everything in, you get the following covariance matrix:

$$ (n\Sigma)_{ij} = n \sqrt{p_i p_j} (\delta_{ij} - \sqrt{p_i p_j}) $$

for $i,j = 1, \ldots, m-1$, which is exactly the covariance matrix in the original answer restricted to its first $m-1$ rows and $m-1$ columns.

This blog entry was my starting point.

$\endgroup$
2
  • 1
    $\begingroup$ Another useful resource is the links provided in: stats.stackexchange.com/questions/2397/… $\endgroup$ Mar 17, 2020 at 5:21
  • 1
    $\begingroup$ Good answer (+1) --- Note that you can embed links with the syntax [textual description](hyperlink). I have taken the liberty of editing this answer to embed your links. $\endgroup$
    – Ben
    Mar 17, 2020 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.