Return to Answer

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ where $O_i$ are observed counts and $E_i$ are expected counts (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

[In the continuous case the Kullback-Leibler divergence is given by $\int _{-\infty }^\infty \, p(x)\,\log {\frac {p(x)}{q(x)}}\,dx$.]

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.

There are a variety of ways you can choose parameter values that will make distributions "close" to some target (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ where $O_i$ are observed counts and $E_i$ are expected counts (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

[In the continuous case the Kullback-Leibler divergence is given by $\int _{-\infty }^\infty \, p(x)\,\log {\frac {p(x)}{q(x)}}\,dx$.]

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ where $O_i$ are observed counts and $E_i$ are expected counts (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

[In the continuous case the Kullback-Leibler divergence is given by $\int _{-\infty }^\infty \, p(x)\,\log {\frac {p(x)}{q(x)}}\,dx$.]

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So that criterion would just be exactly the thing you'd be doing if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

There are a variety of ways you can make distributions close (just as there are a variety of ways of estimating parameters from data).

The most common way would probably be to minimize the Kullback–Leibler divergence

$$\mathrm {KL}(p,q)=\sum _{i}p_i\,\log {\frac {p_i}{q_i}}$$

where $q_i$ would represent the probabilities from your known distribution and $p_i$ would be the one you're "fitting" to it (trying to approximate closely to $q$) -- $p_i$ is therefore a function of parameters.

I'm going to offer an intuitive motivation for it, along the lines hinted at by your post.

Imagine you generated a really large sample (size $N$) from the distribution with the probabiities given by the $q_i$, and wanted to estimate the parameters of the distribution corresponding to the $p_i$ by maximum likelihood. Then with the $q$'s completely specified you'd have a multinomial goodness of fit problem, and with ML as a criterion, you'd be minimizing $-\log\mathcal L = \sum _{{i}}{O_{{i}}\cdot \ln \left({\frac {O_{i}}{E_{i}}}\right)}$ (which you may recognize as half the test statistic in a G-test).

Now if we write $E_i=Nq_i$ and $O_i=N\hat{p}_i$ we get that we're minimizing $\sum _{{i}}{N\hat{p}_{{i}}\cdot \log \left({\frac {N\hat{p}_{i}}{Nq_{i}}}\right)}=N\sum _{{i}}{\hat{p}_{{i}}\cdot \log \left({\frac {\hat{p}_{i}}{q_{i}}}\right)}\propto \sum _{{i}}{\hat{p}_{{i}} \log {\frac {\hat{p}_{i}}{q_{i}}}}$

Now as our sample size $N\to \infty$, $\hat{p}_i \to p_i$, so in the limit (where our sample is our population), we get $\sum _{{i}}{p_{{i}} \log {\frac {p_{i}}{q_{i}}}}$, the K-L divergence.

So minimizing that Kullback-Leibler divergence would just be exactly the thing you'd be doing in the limit if you just generated larger and larger samples and fitted the parameters of your Poisson mixture (say) by maximum likelihood.

You might do other things of course, for example you could do something similar with an ordinary chi-square statistic and try to optimize $\sum _{i}\frac{(p_{i}- q_{i})^2}{q_i}$ or you could take various other goodness of fit criteria and optimize those (e.g. maybe an Anderson-Darling type of criterion), or you might look at moment-matching or quantile-matching or any number of other ways to get a close fit in some sense (by whatever loss function you like) -- and for some particular purposes those various things might be just the ticket.