0
$\begingroup$

I would like an analytical derivation of the Bayesian decision boundary between 2 bivariate uniform distributions. Let me explain with an example. Suppose the two distributions are $$U_1 = 0.25 \text{ if } 0 \le x_1, x_2 \le 2 \text{ else }0$$ and $$U_2 = 0.5 \text{ if } 1 \le x_1 \le 2, 1 \le x2 \le 3 \text{ else }0$$My intuition is the following: the distributions overlap on the square $1 \le x_1,x_2 \le 2$ and $U_2$ is more probable there so it takes it. That is the boundary is $$\text{sample}(x_1,x_2) \in U_2 \text{ if } x1,x2 > 1 $$ So far I have silently assumed that the priors are equal. If that's not the cases then not many things change. The only difference is that if the prior of $U_1$ is greater than $2/3$ it takes the overlapping square. In other words, the boundary becomes $$\text{sample}(x_1,x_2) \in U_2 \text{ if } x1,x2 > 2 $$ How do I prove this though?

PS: My question is kinda the 2d version of this one.

$\endgroup$
4
  • $\begingroup$ Sure. I don't think there's something missing anyway. But I have read that on stackexchange sites is a good practice to add similar questions and clarifying the differences in order to not have them wrongfully flagged as duplicate. Perhaps I should add the similar one in the end? $\endgroup$
    – cgss
    Nov 23, 2020 at 12:23
  • $\begingroup$ Oh yes. I would try to fix it. $\endgroup$
    – cgss
    Nov 23, 2020 at 12:32
  • $\begingroup$ @Xi'an How does it look now? $\endgroup$
    – cgss
    Nov 23, 2020 at 12:38
  • $\begingroup$ @Xi'an Given a sample($x_1, x_2$) the Bayesian classifier labels it as category 1 or category 2 depending on which side of the decision boundary it is. I want to formally solve the problem of finding the equation of said boundary. $\endgroup$
    – cgss
    Nov 23, 2020 at 14:53

1 Answer 1

2
$\begingroup$

Since there is no mention of costs associated with decisions, I assume that the Bayesian decision rule is minimizing the average probability of error.

When Hypothesis $H_0$ is true, the observation $(x_1, x_2)$ is uniformly distributed on a square of side $2$ (sides parallel to the axes) and opposite vertices $(0,0)$ and $(2,2)$.

When Hypothesis $H_1$ is true, the observation $(x_1, x_2)$ is uniformly distributed on a rectangle of area $2$ (sides parallel to the axes) and opposite vertices $(1,1)$ and $(2,3)$.

Thus, the likeliihood ratio $\displaystyle \Lambda(x_1,x_2) = \frac{f_1(x_1,x_2)}{f_0(x_1,x_2)}$ has value $\displaystyle\frac{0}{0.25} = 0$ when $(x_1,x_2)$ lies in an L-shaped region of area $3$, value $\displaystyle\frac{0.5}{0.25} = 2$ when $(x_1,x_2)$ lies in the square of area $1$ with opposite vertices $(1,1)$ and $(2,2)$, and value $\displaystyle\frac{0.5}{0} = \infty$ $(x_1,x_2)$ lies in the square of area $1$ with opposite vertices $(1,2)$ and $(2,2)$. Now, the Bayesian decision rule compares $\Lambda(x_1,x_2)$ to the threshold $\displaystyle \frac{\pi_0}{\pi_1}$ where $\pi_0$ and $\pi_1 = 1-\pi_0$ are the prior probabilities of $H_0$ and $H_1$, and so it is easy to figure out what the Bayesian decision rule is, and what the corresponding decision boundary is.

Let $\Gamma_0$ and $\Gamma_1$ denote the disjoint regions such that when $(x_1,x_2) \in \Gamma_i$, the decision is that $\Gamma_i$ is the true hypothesis.

  • If $\displaystyle\pi_0 < \frac 23$ so that $\displaystyle\frac{\pi_0}{\pi_1} < 2$, $\Lambda(x_1,x_2) > \displaystyle\frac{\pi_0}{\pi_1}$ whenever $(x_1,x_2)$ lies in the rectangular region of area $2$ (sides parallel to the axes) and opposite vertices $(1,1)$ and $(2,3)$. Thus, $\Gamma_1$ is this rectangular region, and we have \begin{align}\Gamma_1 &= \{(x_1,x_2)\colon x_1 \geq 1, x_2 \geq 1\}\\ \Gamma_0 &= \{(x_1,x_2)\colon x_1 < 1~ \textbf{or} ~ x_2 < 1\}\end{align}

  • If $\displaystyle\pi_0 > \frac 23$ so that $\displaystyle\frac{\pi_0}{\pi_1} > 2$, $\Lambda(x_1,x_2) > \displaystyle\frac{\pi_0}{\pi_1}$ whenever $(x_1,x_2)$ lies in the square region of area $1$ (sides parallel to the axes) and opposite vertices $(1,2)$ and $(2,3)$. Thus, $\Gamma_1$ is this square region, and we have \begin{align}\Gamma_1 &= \{(x_1,x_2)\colon x_1 \geq 1, x_2 \geq 2\}\\ \Gamma_0 &= \{(x_1,x_2)\colon x_1 < 2~, x_2 <2\}\end{align}

  • If $\pi_0 = \frac 23$ exactly so that $\Lambda(x_1,x_2)$ has value $2$ exactly equal to the threshold for all $(x_1,x_2)$ in the square region of area $1$ with opposite vertices $(1,1)$ and $(2,2)$. In this instance, the average error probability is the same regardless of which points in this region we assign to $\Gamma_0$ and which we assign to $\Gamma_1$ !! So, we arbitrarily choose to assign them all to $\Gamma_0$ resulting in \begin{align}\Gamma_1 &= \{(x_1,x_2)\colon x_1 \geq 1, x_2 \geq 2\}\\ \Gamma_0 &= \{(x_1,x_2)\colon x_1 < 2~, x_2 <2\}\end{align}

If the OP desires, he can massage the above information into a single humongous formula that upon plugging in the values of $\pi_0, \pi_1$ gives the exact decision boundaries directly and thereby amaze his friends and TA.

$\endgroup$
1
  • $\begingroup$ I was trying to use the rule $U_1\cdot \text{ prior of } U_1 = U_2\cdot \text{ prior of } U_2$. I understand you are using the same rule before cross multiplying, correct? You are using the boundary as known and figuring out the decision. I agree. The problem is I am trying to have the boundary as unknown and the solve the equation to find what I see in my drawings. It's like when we find out the intersection point in the 1d case. Does it makes sense what I am asking? $\endgroup$
    – cgss
    Nov 23, 2020 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.