Skip to main content
4 of 6
added 815 characters in body
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.

Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k