2
$\begingroup$

This question is concerning a similar problem as mentioned in this question. The only difference is that in my case the variances are unequal.

To recap, consider a two class scenario. At the decision boundary, the posterior probability of classifying a data point into two classes will be equal i.e. $p(y=1|x) = p(y=2|x)$

Posterior definition

$p(y=1|x) = \frac{p(x|y=1) * P(y=1)}{p(x)}$

$p(y=2|x) = \frac{p(x|y=2) * P(y=2)}{p(x)}$

where likelihoods are Gaussian i.e.

$p(x|y=1) = \mathcal{N}(x|\mu_1, \sigma_1)$

$p(x|y=2) = \mathcal{N}(x|\mu_2, \sigma_2)$

So at the decision boundary, $p(y=1|x^*) = p(y=2|x^*)$ where $x^*$ is the threshold

$ \begin{align} &p(y=1|x^*) = p(y=2|x^*) \\ &\Longrightarrow\frac{1}{\sqrt{2\pi\sigma_1^2}}\exp(-\frac{(x^* - \mu_1)^2}{2\sigma_1^2}) * P(y=1) = \frac{1}{\sqrt{2\pi\sigma_2^2}}\exp(-\frac{(x^* - \mu_2)^2}{2\sigma_2^2})* P(y=2)\\ \end{align} $

Taking log on both sides,

$ \begin{align} \Rightarrow & \small-\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\log\sqrt{2\pi}\sigma_1 + \log P(y=1)= -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} -\log\sqrt{2\pi}\sigma_2 + \log P(y=2)\\ \end{align} $

To get the threshold, we would solve for $x^*$ in the above equation. But since the variances are unequal this will remain a quadratic equation and hence it is possible to get complex values for $x^*$.

If this is the case, then what does it mean to have a complex threshold?

Further context:

I'm using a 2 component Gaussian mixture model and planning to find the threshold to create a mask as specified in the paper snippet below. As such, I was expecting the threshold to be real in order to create the mask. Hence my confusion.

enter image description here

$\endgroup$
4
  • $\begingroup$ It isn't possible to get complex values. The boundary is circular. $\endgroup$
    – whuber
    Jul 6, 2021 at 22:10
  • $\begingroup$ So the root of the quadratic equation i.e. $x^*$ is guaranteed to be real? $\endgroup$
    – ashnair1
    Jul 7, 2021 at 10:53
  • 1
    $\begingroup$ Yes. Geometry makes this obvious: the boundary occurs where the ratio of the distances to the two centers is a constant. Such loci are circles. (When that ratio is $1,$ the locus is the perpendicular bisector of the centers, which is a generalized circle of infinite radius; when the ratio is $0,$ the locus reduces to a point: a circle of zero radius.) $\endgroup$
    – whuber
    Jul 7, 2021 at 13:47
  • 1
    $\begingroup$ When the variances differ and you are in two dimensions, the boundary is elliptical: see the first two figures in my post at stats.stackexchange.com/a/81952/919. It is possible for the set of solutions to be empty. $\endgroup$
    – whuber
    Jul 8, 2021 at 14:14

2 Answers 2

0
$\begingroup$

Verification attempt via quadratic discriminant:

To check the nature of roots of this equation, we could check the discriminant and confirm if it's greater than or equal to 0 for real solutions and less than 0 for complex ones.

Consider the previous equation
$$ \begin{align} \small-\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\log\sqrt{2\pi}\sigma_1 + \log P(y=1)= -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} -\log\sqrt{2\pi}\sigma_2 + \log P(y=2)\\ \end{align} $$Rearranging a bit we get,$$ \begin{align} \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2}+\log\sqrt{2\pi}\sigma_1 -\log\sqrt{2\pi}\sigma_2 + \log P(y=2) -\log P(y=1)=0\\ \end{align} $$Group the log terms,$$ \begin{align} \rightarrow \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} + \log \frac{\sigma_1}{\sigma_2} + \log \frac{P(y=2)}{P(y=1)} = 0\\ \rightarrow \small\frac{(x^* - \mu_1)^2}{2\sigma_1^2} -\frac{(x^* - \mu_2)^2}{2\sigma_2^2} + \log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)} = 0 \end{align} $$Let's now only consider the first two terms and for convenience set $\log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)} = k$ a constant. Expanding the squares we get,$$ \begin{align} \frac{2(\sigma_2^2 - \sigma_1^2 )x^{*^2} -4(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 2(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)}{4\sigma_1^2\sigma_2^2} +k=0\\ \frac{0.5(\sigma_2^2 - \sigma_1^2 )x^{*^2} -(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 0.5(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)}{\sigma_1^2\sigma_2^2} +k=0\\ 0.5(\sigma_2^2 - \sigma_1^2 )x^{*^2} -(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)x^* + 0.5(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2) + k\sigma_1^2\sigma_2^2=0 \end{align} $$ For a quadratic equation of the form $ax^2 + bx + c = 0$, the discriminant ($\Delta$) $= b^2 - 4ac$

From (6) we calculate the individual terms,
$$ \begin{align} b^2 =\left(-(\sigma_2^2\mu_1 - \sigma_1^2\mu_2)\right) ^2= \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2\\ 4ac =4 * 0.5(\sigma_2^2 - \sigma_1^2 )* \left(0.5*(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2)+ k\sigma_1^2\sigma_2^2 \right)\\ = 4 * 0.5(\sigma_2^2 - \sigma_1^2 )*0.5* \left(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2+ 2k\sigma_1^2\sigma_2^2 \right)\\ = (\sigma_2^2 - \sigma_1^2 ) \left(\sigma_2^2\mu_1^2 - \sigma_1^2\mu_2^2+ 2k\sigma_1^2\sigma_2^2 \right)\\ = (\sigma_1^2 - \sigma_2^2 ) \left(\sigma_1^2\mu_2^2-\sigma_2^2\mu_1^2 - 2k\sigma_1^2\sigma_2^2 \right)\\ \end{align} $$Calculate discriminant $$ \begin{align} b^2 - 4ac = \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2 - (\sigma_1^2 - \sigma_2^2 ) \left(\sigma_1^2\mu_2^2-\sigma_2^2\mu_1^2 - 2k\sigma_1^2\sigma_2^2 \right)\\ = \left(\sigma_1^2\mu_2 - \sigma_2^2\mu_1\right) ^2 - \left(\sigma_1^4\mu_2^2 - \sigma_1^2\sigma_2^2\mu_1^2-2k\sigma_1^4\sigma_2^2 - \sigma_1^2\sigma_2^2\mu_2^2 + \sigma_2^4\mu_1^2 + 2k\sigma_1^2\sigma_2^4\right)\\ = \left(\sigma_1^4\mu_2^2 -2\sigma_1^2\sigma_2^2\mu_1\mu_2 +\sigma_2^4\mu_1^2\right) - \left(\sigma_1^4\mu_2^2 - \sigma_1^2\sigma_2^2\mu_1^2-2k\sigma_1^4\sigma_2^2 - \sigma_1^2\sigma_2^2\mu_2^2 + \sigma_2^4\mu_1^2 + 2k\sigma_1^2\sigma_2^4\right)\\ = \sigma_1^2\sigma_2^2\left(\mu_1^2 +\mu_2^2 - 2\mu_1\mu_2 + 2k(\sigma_1^2 - \sigma_2^2)\right)\\ = \sigma_1^2\sigma_2^2\left( \left(\mu_1 - \mu_2\right)^2 + 2k(\sigma_1^2 - \sigma_2^2)\right)\\ \end{align} $$

So the discriminant $\Delta =\sigma_1^2\sigma_2^2\left(\left(\mu_1-\mu_2\right)^2 + 2\log \frac{\sigma_1P(y=2)}{\sigma_2P(y=1)}(\sigma_1^2 - \sigma_2^2)\right)$.

It can be seen that since the discriminant is strictly non-negative, it's not possible for complex solutions to arise.

Edit: This is incorrect. As pointed out by @whuber in the comments, setting $P(y=2)$ to a low value would cause $\Delta$ to be negative.

$\endgroup$
7
  • $\begingroup$ (1) You really don't have to work this hard. At the outset, change to the variable $y = (x-\mu_1)/\sigma_1$ and 90% of your algebra will be unnecessary. (2) Everything comes down to the unproven assertion that $\Delta \gt 0.$ Unfortunately, that is generally false, because by making $P(y=2)$ sufficiently small, $\Delta$ can be made negative. $\endgroup$
    – whuber
    Jul 8, 2021 at 14:20
  • $\begingroup$ My bad. You're right. But if the threshold ($x^*$) can't be complex why is it possible for $\Delta$ to be negative? Shouldn't the condition $\Delta >=0$ hold for real thresholds? Either I'm missing something obvious here or my calculation is incorrect. $\endgroup$
    – ashnair1
    Jul 8, 2021 at 15:24
  • 1
    $\begingroup$ There's no such thing as a complex threshold: one therefore looks only for real solutions. It is possible for there to be no real solutions, in which case there is no threshold: you would conclude there is just one class. $\endgroup$
    – whuber
    Jul 8, 2021 at 15:52
  • $\begingroup$ So it is possible for roots to be complex! But the implication is that there is no decision surface. $\endgroup$
    – ashnair1
    Jul 8, 2021 at 15:59
  • 1
    $\begingroup$ The only roots one looks for are real. Complex roots, whether they might exist or not, are not relevant. I apologize for misleading you. $\endgroup$
    – whuber
    Jul 8, 2021 at 16:29
0
$\begingroup$

The solution x* will be two real values.

This sort of decision boundary is typical in mixture models, for instance $\sigma_1^2 = 1$ is a "typical" scenario, and $\sigma_2^2 = 100$ is an "outlier" scenario.

The low variance, highly peeked distribution is selected around its mean; elsewhere above and below this region attributed to the low variance distribution, the diffuse high variance distribution is selected.

West & Harrison discuss models involving "typical" and "outlier" observation variance $V$, in their notation $V_\text{typical}=1$ and $V_\text{outlier} = 100$.

  • West, Mike, and Jeff Harrison. Bayesian forecasting and dynamic models. Springer Science & Business Media, 2006.
$\endgroup$
4
  • 1
    $\begingroup$ The assertion on which this post is based -- there will be two real values -- is not correct in all cases. $\endgroup$
    – whuber
    Jul 8, 2021 at 15:53
  • $\begingroup$ @whuber - please provide a counter example relevant to the OP's case, "variances are unequal", $\sigma_2^2 > \sigma_1^2 > 0$. $\endgroup$
    – krkeane
    Jul 8, 2021 at 17:07
  • $\begingroup$ See my first comment to another answer in this thread. $\endgroup$
    – whuber
    Jul 8, 2021 at 17:51
  • 1
    $\begingroup$ In a mixture of "typical" and "outlier" observations, often $P(\text{typical}) \gg P(\text{outlier})$. In the opposite scenario (corresponding to the point made by @whuber above), when a relatively more precise distribution is a "small enough" component of a mixture, the posterior probability of the precise distribution may be always less than ("below") the posterior probability density of the more diffuse distribution. $\endgroup$
    – krkeane
    Jul 9, 2021 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.