# Can Gaussian mixture models with different amount of populations have the same pdf

Concretely, denote $$f_{\mu, \sigma}$$ for the pdf of a normally distributed random variable with mean $$\mu$$ and standard deviation $$\sigma$$. Is it possible that

$$\displaystyle \sum_{i = 0}^n p_i f_{\mu_{1i}, \sigma_{1i}}(x) = \displaystyle \sum_{j = 0}^k q_j f_{\mu_{2j}, \sigma_{2j}}(x) \ \ \ \ \ \sum p_i = \sum q_i = 1; p_i, q_i > 0$$

for $$n \neq k$$? Intuitively, I'd say yes because if two normals have the same mean, the weighted sum looks like a bell curve with the same mean and different standard deviation, but I have not been able to proof it.

• It's unclear what you are asking, for several reasons. First, your argument applies to sums of random variables, not to mixture distributions, so are you sure you are asking about mixtures? (This is a crucial ambiguity to resolve.) Second, you use identical notations for the distributions: is that intended, or do you posit one set of components for one sum and another potentially different set for the second? Third, are you supposing all the components differ from each other or not? Fourth, are you assuming all the $p_i$ and $q_i$ are nonzero?
– whuber
Commented Feb 28, 2020 at 17:44
• @whuber Mixtures is what I meant. Why does it not hold for mixtures? Second, that was not intended, the distributions left and right can be different, I fixed the notation. Third, yes. Fourth, yes. Thanks for taking the time to think about my question. Commented Feb 28, 2020 at 17:57
• @whuber or well, not all components different, but I am not interested in the case $p_1 f_{\mu, \sigma} + p_2 f_{\mu, \sigma} = (p_1 + p_2)f_{\mu, \sigma}$. Commented Feb 28, 2020 at 18:01
• Thank you for the clarifications! Using the techniques at stats.stackexchange.com/questions/429868 (and very nearly the same argument), you can show the answer is in the negative: when the two sums are equal, they must both be expressible in identical ways as two Normal mixtures up to the inconsequential differences created by allowing some coefficients to be zero, allowing some of the components to be equal, or re-ordering the indexes.
– whuber
Commented Feb 28, 2020 at 18:07

The answer is no: the mixture determines its components uniquely.

There's an elementary way to show it. The idea is that each component of the mixture determines how the density behaves at extreme values and we can use this asymptotic behavior to identify (and remove) the components one by one.

To carry out this program, consider a Normal mixture

$$f(x) = \sum_{i=1}^n \frac{p_i}{\sigma_i} \phi\left(\frac{x-\mu_i}{\sigma_i}\right)$$

where

$$\phi(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}$$

is the standard Normal density, the $$p_i\ne 0$$ and $$\mu_i$$ are any real numbers, the $$\sigma_i$$ are positive real numbers, and the ordered pairs $$(\mu_i,\sigma_i)$$ are all distinct. Re-order the indexes $$i$$ if necessary so that

$$\sigma_1 \le \sigma_2 \le \ldots \le \sigma_n$$

and, whenever $$\sigma_i = \sigma_j,$$ then $$\mu_i \le \mu_j.$$ This ordering (the lexicographic ordering) is unique. We're going to examine the last component first (to reduce the length of the sum) and to that end this ordering has us first visit the most spread-out components and, within them, those that are positioned at higher values.

To study what happens to $$f$$ as $$x$$ grows large, consider

\eqalign{&\frac{\sigma_n}{\phi\left(\frac{x-\mu_n}{\sigma_n}\right)} f(x) \\&= p_n + \sum_{i=1}^{n-1} p_i \frac{\sigma_n}{\sigma_i} \exp\left(\frac{(\sigma_i^2-\sigma_n^2)x^2 - 2x\left(\sigma_n^2\mu_i - \sigma_i^2\mu_n)\right) - \sigma_n^2\mu_i^2 + \sigma_i^2\mu_n^2}{2\sigma_i^2\sigma_n^2}\right). }

Because every $$\sigma_i^2-\sigma_n^2 \le 0,$$ the limiting values of the exponentials in the sum are all $$0$$ unless the coefficient of the $$x^2$$ term is zero, which is only the case when $$\sigma_i = \sigma_n.$$ But in those cases the limiting values are still zero unless the coefficient of $$x$$ is zero, which (since now $$\sigma_i^2=\sigma_n^2$$) occurs only when $$\mu_i=\mu_n.$$ But we have arranged at the outset that this never happens: there is no $$i\ne n$$ for which $$(\mu_i,\sigma_i)=(\mu_n,\sigma_n).$$ Thus,

$$\lim_{x\to\infty} \frac{\sigma_n}{\phi\left(\frac{x-\mu_n}{\sigma_n}\right)} f(x) = p_n.$$

Had we used any value other than $$\sigma_n$$ in this analysis, the limit would have been either $$0$$ or diverging to $$\pm \infty;$$ and, having used $$\sigma_n,$$ using any other value in place of $$\mu_n$$ also would have produced a limit of $$0$$ or $$\pm\infty.$$ In other words, $$(\mu_n,\sigma_n)$$ is the only parameter for which we can achieve a finite nonzero limit and that limit determines $$p_n.$$

This shows that any mixture of $$n$$ distinct Normal densities determines its last component (in the lexicographic ordering of components). Subtracting off this component yields a Normal mixture with one less component, instantly giving an inductive proof of the result:

Let $$p_i\ne 0,q_j\ne 0,\sigma_i\gt 0,\tau_j\gt 0,\mu_i,$$ and $$\nu_j$$ be any real numbers for $$1\le i \le n$$ and $$1\le j \le k.$$ If every number $$x$$ in a set with no upper bound satisfies $$\sum_{i=1}^n \frac{p_i}{\sigma_i} \phi\left(\frac{x-\mu_i}{\sigma_i}\right) = \sum_{i=1}^k \frac{q_i}{\tau_i} \phi\left(\frac{x-\nu_i}{\tau_i}\right)$$ and each sum involves distinct Normal components and is ordered lexicographically, then $$n=k$$ and for each $$1\le i\le n,$$ $$p_i=q_i,$$ $$\mu_i=\nu_i,$$ and $$\sigma_i=\tau_i.$$

(I phrased this in a way that indicates how this analysis can be generalized to some other families of distributions, including discrete distributions.)

Notice we never had to assume the $$p_i$$ were positive or that they had to sum to unity. In fact, we didn't even have to assume $$n$$ and $$k$$ are finite! In the countably infinite case, all that's needed to carry the demonstration through is that the sets of $$\sigma_i$$ and $$\tau_j$$ are bounded above, have only one accumulation point each, and that each set of $$\mu_i$$ and $$\nu_j$$ has at most one accumulation point in the extended reals.

• is there a textbook citation of this anywhere? Commented Dec 9, 2022 at 22:00
• @JimmyTwoCents I haven't researched that.
– whuber
Commented Dec 9, 2022 at 22:06
• @Jimmy A couple of years later, Christian Hennig identified a reference in a post at stats.stackexchange.com/a/594181/919.
– whuber
Commented Sep 5, 2023 at 14:16
• Nice proof! Cheney & Light, Lemma 1 in Chap 13 state and prove a similar result for the simplest exponential case. I guess that we can similarly claim that a finite family of Gaussian densities with distinct parameters are linearly independent functions on any domain that has a point of accumulation.
– Yves
Commented Nov 8, 2023 at 7:32
• @Andyale As I wrote in a comment to your related question, because the multidimensional Normal distribution is one in which all linear combinations are univariate Normal, there's nothing more to be shown in the multidimensional case.
– whuber
Commented Dec 9, 2023 at 17:48