# Decision boundary between two Gaussians of unequal variance

This question is concerning a similar problem as mentioned in this question. The only difference is that in my case the variances are unequal.

To recap, consider a two class scenario. At the decision boundary, the posterior probability of classifying a data point into two classes will be equal i.e. $$p(y=1|x) = p(y=2|x)$$

Posterior definition

$$p(y=1|x) = \frac{p(x|y=1) * P(y=1)}{p(x)}$$

$$p(y=2|x) = \frac{p(x|y=2) * P(y=2)}{p(x)}$$

where likelihoods are Gaussian i.e.

$$p(x|y=1) = \mathcal{N}(x|\mu_1, \sigma_1)$$

$$p(x|y=2) = \mathcal{N}(x|\mu_2, \sigma_2)$$

So at the decision boundary, $$p(y=1|x^*) = p(y=2|x^*)$$ where $$x^*$$ is the threshold

\begin{align} &p(y=1|x^*) = p(y=2|x^*) \\ &\Longrightarrow\frac{1}{\sqrt{2\pi\sigma_1^2}}\exp(-\frac{(x^* - \mu_1)^2}{2\sigma_1^2}) * P(y=1) = \frac{1}{\sqrt{2\pi\sigma_2^2}}\exp(-\frac{(x^* - \mu_2)^2}{2\sigma_2^2})* P(y=2)\\ \end{align}

Taking log on both sides,

\begin{align} \Rightarrow & \small-\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\log\sqrt{2\pi}\sigma_1 + \log P(y=1)= -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} -\log\sqrt{2\pi}\sigma_2 + \log P(y=2)\\ \end{align}

To get the threshold, we would solve for $$x^*$$ in the above equation. But since the variances are unequal this will remain a quadratic equation and hence it is possible to get complex values for $$x^*$$.

If this is the case, then what does it mean to have a complex threshold?

#### Further context:

I'm using a 2 component Gaussian mixture model and planning to find the threshold to create a mask as specified in the paper snippet below. As such, I was expecting the threshold to be real in order to create the mask. Hence my confusion.

• It isn't possible to get complex values. The boundary is circular.
– whuber
Jul 6, 2021 at 22:10
• So the root of the quadratic equation i.e. $x^*$ is guaranteed to be real? Jul 7, 2021 at 10:53
• Yes. Geometry makes this obvious: the boundary occurs where the ratio of the distances to the two centers is a constant. Such loci are circles. (When that ratio is $1,$ the locus is the perpendicular bisector of the centers, which is a generalized circle of infinite radius; when the ratio is $0,$ the locus reduces to a point: a circle of zero radius.)
– whuber
Jul 7, 2021 at 13:47
• When the variances differ and you are in two dimensions, the boundary is elliptical: see the first two figures in my post at stats.stackexchange.com/a/81952/919. It is possible for the set of solutions to be empty.
– whuber
Jul 8, 2021 at 14:14

#### Verification attempt via quadratic discriminant:

To check the nature of roots of this equation, we could check the discriminant and confirm if it's greater than or equal to 0 for real solutions and less than 0 for complex ones.

Consider the previous equation
\begin{align} \small-\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\log\sqrt{2\pi}\sigma_1 + \log P(y=1)= -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} -\log\sqrt{2\pi}\sigma_2 + \log P(y=2)\\ \end{align}Rearranging a bit we get,\begin{align} \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2}+\log\sqrt{2\pi}\sigma_1 -\log\sqrt{2\pi}\sigma_2 + \log P(y=2) -\log P(y=1)=0\\ \end{align}Group the log terms,\begin{align} \rightarrow \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} + \log \frac{\sigma_1}{\sigma_2} + \log \frac{P(y=2)}{P(y=1)} = 0\\ \rightarrow \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} + \log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)} = 0 \end{align}Let's now only consider the first two terms and for convenience set $$\log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)} = k$$ a constant. Expanding the squares we get,\begin{align} \frac{2(\sigma_2^2 - \sigma_1^2 )x^{*^2} -4(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 2(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)}{4\sigma_1^2\sigma_2^2} +k=0\\ \frac{0.5(\sigma_2^2 - \sigma_1^2 )x^{*^2} -(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 0.5(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)}{\sigma_1^2\sigma_2^2} +k=0\\ 0.5(\sigma_2^2 - \sigma_1^2 )x^{*^2} -(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 0.5(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2) + k\sigma_1^2\sigma_2^2=0 \end{align} For a quadratic equation of the form $$ax^2 + bx + c = 0$$, the discriminant ($$\Delta$$) $$= b^2 - 4ac$$

From (6) we calculate the individual terms,
\begin{align} b^2 =\left(-(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)\right) ^2= \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2\\ 4ac =4 * 0.5(\sigma_2^2 - \sigma_1^2 )* \left(0.5*(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)+ k\sigma_1^2\sigma_2^2 \right)\\ = 4 * 0.5(\sigma_2^2 - \sigma_1^2 )*0.5* \left(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2+ 2k\sigma_1^2\sigma_2^2 \right)\\ = (\sigma_2^2 - \sigma_1^2 ) \left(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2+ 2k\sigma_1^2\sigma_2^2 \right)\\ = (\sigma_1^2 - \sigma_2^2 ) \left(\sigma_1^2\mu_2^2-\sigma_2^2\mu_1^2 - 2k\sigma_1^2\sigma_2^2 \right)\\ \end{align}Calculate discriminant \begin{align} b^2 - 4ac = \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2 - (\sigma_1^2 - \sigma_2^2 ) \left(\sigma_1^2\mu_2^2-\sigma_2^2\mu_1^2 - 2k\sigma_1^2\sigma_2^2 \right)\\ = \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2 - \left(\sigma_1^4\mu_2^2 - \sigma_1^2\sigma_2^2\mu_1^2-2k\sigma_1^4\sigma_2^2 - \sigma_1^2\sigma_2^2\mu_2^2 + \sigma_2^4\mu_1^2 + 2k\sigma_1^2\sigma_2^4\right)\\ = \left(\sigma_1^4\mu_2^2 -2\sigma_1^2\sigma_2^2\mu_1\mu_2 +\sigma_2^4\mu_1^2\right) - \left(\sigma_1^4\mu_2^2 - \sigma_1^2\sigma_2^2\mu_1^2-2k\sigma_1^4\sigma_2^2 - \sigma_1^2\sigma_2^2\mu_2^2 + \sigma_2^4\mu_1^2 + 2k\sigma_1^2\sigma_2^4\right)\\ = \sigma_1^2\sigma_2^2\left(\mu_1^2 +\mu_2^2 - 2\mu_1\mu_2 + 2k(\sigma_1^2 - \sigma_2^2)\right)\\ = \sigma_1^2\sigma_2^2\left( \left(\mu_1 - \mu_2\right)^2 + 2k(\sigma_1^2 - \sigma_2^2)\right)\\ \end{align}

So the discriminant $$\Delta =\sigma_1^2\sigma_2^2\left(\left(\mu_1-\mu_2\right)^2 + 2\log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)}(\sigma_1^2 - \sigma_2^2)\right)$$.

It can be seen that since the discriminant is strictly non-negative, it's not possible for complex solutions to arise.

Edit: This is incorrect. As pointed out by @whuber in the comments, setting $$P(y=2)$$ to a low value would cause $$\Delta$$ to be negative.

• (1) You really don't have to work this hard. At the outset, change to the variable $y = (x-\mu_1)/\sigma_1$ and 90% of your algebra will be unnecessary. (2) Everything comes down to the unproven assertion that $\Delta \gt 0.$ Unfortunately, that is generally false, because by making $P(y=2)$ sufficiently small, $\Delta$ can be made negative.
– whuber
Jul 8, 2021 at 14:20
• My bad. You're right. But if the threshold ($x^*$) can't be complex why is it possible for $\Delta$ to be negative? Shouldn't the condition $\Delta >=0$ hold for real thresholds? Either I'm missing something obvious here or my calculation is incorrect. Jul 8, 2021 at 15:24
• There's no such thing as a complex threshold: one therefore looks only for real solutions. It is possible for there to be no real solutions, in which case there is no threshold: you would conclude there is just one class.
– whuber
Jul 8, 2021 at 15:52
• So it is possible for roots to be complex! But the implication is that there is no decision surface. Jul 8, 2021 at 15:59
• The only roots one looks for are real. Complex roots, whether they might exist or not, are not relevant. I apologize for misleading you.
– whuber
Jul 8, 2021 at 16:29

The solution x* will be two real values.

This sort of decision boundary is typical in mixture models, for instance $$\sigma_1^2 = 1$$ is a "typical" scenario, and $$\sigma_2^2 = 100$$ is an "outlier" scenario.

The low variance, highly peeked distribution is selected around its mean; elsewhere above and below this region attributed to the low variance distribution, the diffuse high variance distribution is selected.

West & Harrison discuss models involving "typical" and "outlier" observation variance $$V$$, in their notation $$V_\text{typical}=1$$ and $$V_\text{outlier} = 100$$.

• West, Mike, and Jeff Harrison. Bayesian forecasting and dynamic models. Springer Science & Business Media, 2006.
• The assertion on which this post is based -- there will be two real values -- is not correct in all cases.
– whuber
Jul 8, 2021 at 15:53
• @whuber - please provide a counter example relevant to the OP's case, "variances are unequal", $\sigma_2^2 > \sigma_1^2 > 0$. Jul 8, 2021 at 17:07
• – whuber
Jul 8, 2021 at 17:51
• In a mixture of "typical" and "outlier" observations, often $P(\text{typical}) \gg P(\text{outlier})$. In the opposite scenario (corresponding to the point made by @whuber above), when a relatively more precise distribution is a "small enough" component of a mixture, the posterior probability of the precise distribution may be always less than ("below") the posterior probability density of the more diffuse distribution. Jul 9, 2021 at 10:23