Let $(X,Y): \Omega \to (\mathbb{R}^d, F)$, with $F = \left\{ 1, \dots, n \right\}$ be a random variable. According to Wikipedia the Bayes classifier is $$C_B(x) = \arg\, \max \mathbb{P}(Y= i \,|\, X=x),$$ with $x \in \mathbb{R}^d$. Now I understand the idea, "pick the most likely one". However, if $X$ has a density, then $X=x$ is a zero probability event. How should conditioning on this event be understood?

Of course you can define something like $$\mathbb{P}(Y=i \,|\, X=x) = \lim \limits_{\varepsilon \to 0} \mathbb{P}(Y=i \, | \, X \in B(x, \varepsilon)),$$ where $B(x,\varepsilon)$ is a ball of radius $\varepsilon > 0$ around $x$, but I'm unable to find a reference for this.

  • 1
    $\begingroup$ While $\mathbb{P}(X=x)=0$, the conditional distribution of $Y$ given $X=x$ is uniquely defined almost everywhere. $\endgroup$
    – Xi'an
    Commented Nov 15, 2016 at 13:37
  • 2
    $\begingroup$ $P(Y=i \vert X=x)$ is to be interpreted as having observed $x$, what is the probability that it comes from class $i$. It is computed from $f(x\vert Y=i)p(Y=i)$, where we assume that the class conditional densities $f(x\vert Y=i)$ and prior probs $p(Y=i)$ are known in advance. $\endgroup$
    – Lella
    Commented Nov 15, 2016 at 14:02
  • $\begingroup$ Recent related question: stats.stackexchange.com/questions/246009/… $\endgroup$
    – Tim
    Commented Nov 15, 2016 at 14:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.