# Function proportional to the log likelihood for the Gaussian distribution

The following question is crossposted from MathStackExchange upon recommendation from the MSE community and a lack of responses on my post over there.

Consider the following problem from a course on statistical inference:

If we generate a sample $$x_i$$ for $$i \in$$ {$$1 ... n$$} from $$p(x_i) = \sum_{k=1}^2 w_kp(x_i| \mu _k, \sigma^2_k)$$ where $$p(x_i| \mu _k, \sigma ^2_k)$$ are Gaussian densities and $$w_2 = 1 - w_1$$ (where we assume that all the parameters are unknown)

Find the log-likelihood function for the sample $$x_{1:n}$$.

The log-likelihood is defined as:

$$l( \theta | x_{1:n}) = \log(p(x_{1:n})) = \log( \Pi _{i=1}^n p(x_i)) = \sum _{i=1}^n \log(p(x_i))$$

Now substituting in for $$p(x_i)$$ we get:

$$l( \theta | x_{1:n}) = \sum_{i=1}^n \log \Big{(} \sum_{k=1}^2 w_k p(x_i | \mu _k , \sigma^2_k) \Big{)}$$

We can now substitute in the Gaussian densities which gives us:

$$l( \theta | x_{1:n}) = \sum_{i=1}^n \log \Big{[} w_1(2 \pi \sigma ^2 _1)^{-1/2} \exp \Big{(}\frac{(x - \mu _1)^2}{2 \sigma ^2 _1} \Big{)} + w_2 (2 \pi \sigma ^2 _2)^{-1/2} \exp \Big{(}\frac{(x - \mu _2)^2}{2 \sigma ^2 _2} \Big{)} \Big{]}$$

The following solution aligns with my working, however, there is a proportionality relation on the last line that is unclear to me.

Why does the final line hold? I can't see a clear and obvious way to get from the point at which I have substituted the Gaussian densities into the sum to it being proportional to the given double summation.

I understand that we can exclude any multiplicative constants, however, it seems as though the summation has been taken out of the logarithm at some stage in order to derive a proportionality relation. Although I'm not sure if this holds and if it does, why does it hold?

Edit

It seems as though this is a mistake as Whuber presents a counterexample in the comments. Here is the second part of the question:

Presumably, this is also incorrect for similar reasons to the above. If this does end up being incorrect is there any way to salvage this?

• You're right. The last line is clearly incorrect: it treats the log as if it distributed over summation. Even in the simplest case $n=1$ it's wrong. For a counterexample take $w_k=1/2,$ $\sigma_k=1,$ and $\mu_2=-\mu_1\gg 1$ and graph the two functions of $x_1$ to show they are not equal. The alleged value is manifestly quadratic, whence it has a unique local maximum (regardless of any constant of proportionality), whereas the log likelihood itself will be bimodal.
– whuber
Commented May 15, 2023 at 20:13
• The second part of the question seems to build on the first part though. Does that mean the entire EM algorithm is applied incorrectly? I have attached a screenshot of the continuation at the bottom of my post @whuber Commented May 15, 2023 at 20:24
• The mistake essentially is fitting a Gaussian model (not a two-component Gaussian mixture) that is overparameterized (two effective parameters are determined by the five numbers $\mu_k,$ $\sigma_k,$ and $w_1$). Apart from some extraordinary coincidence in the algorithm, then, this should never converge: the parameter estimates will wander around nearer and nearer to a three-dimensional manifold of solutions all of which give the same likelihoods.
– whuber
Commented May 15, 2023 at 20:46
• The second part of the question seems correct to me but it doesn't seem to build on top of the first part. Note that this second part if based on the log-likelihood of both x_i and z_i while the first part of the question is only based on the log-likelihood of the x_i. Commented May 17, 2023 at 6:48