4
$\begingroup$

Let $X$ be a random variable with values in $\mathcal{X}$ and $Y$ a random variable with values in $\{0,1\}$. Is it true that : $$\mathbb{P}(g(X) \ne Y) = \int_\mathcal{X} \mathbb{P}\big(g(X)\ne Y \hspace{1mm} \mid X=x\big)d\mathbb{P}_X(x)$$ ?

I agree with this equality in the case where $X$ is discrete (i.e $\mathcal{X}$ is a countable set), but I can't prove why it holds in the general setting where $X$ is not discrete and does not have a density function. Actually, I can't prove the equality when $X$ has a density neither.

I came across this equality in at least 2 articles about statistical learning theory :

The statements, whose proof uses the equality above, say that the Bayes classifier realizes the minimum of the risk $\mathbb{P}(g(X)\ne Y)$ over all measurable functions $g$.

|cite|improve this question
$\endgroup$
1
$\begingroup$

There's no need to treat separately the cases where $X$ is discrete, continuous or neither. We are just taking an iterated expectation here, which is always permissible. The idea is that since probabilities are simply expectations of indicator random variables we can always write something like

\begin{align} P(g(X) \neq Y) &= \text{E} \left (I_{ \{ g(X) \neq Y \}} \right ) \\ &= \text{E} \left [ \text{E} \left (I_{ \{ g(X) \neq Y \}} \mid X \right ) \right ] \\ &= \text{E}[ P(g(X) \neq Y \mid X)] \end{align}

and the integral above is just another way of writing this down. Again, this holds true no matter what "type" of random variable $X$ happens to be.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ You are using $1_{\{g(X)\ne Y\} }= E ( 1_{\{g(X)\ne Y\}} / X)$. Is it because you consider that $1_{\{g(X)\ne Y\}}$ is $\sigma(X)$-measurable ? $\endgroup$ – dada Sep 25 '16 at 19:06
  • $\begingroup$ OK I got it. You didn't use the equality $1_{\{g(X)\ne Y\}} = E(1_{\{g(X)\ne Y\}}/X)$. You actually used $E(1_{\{g(X)\ne Y\}})=E [ E(1_{\{g(X)\ne Y\}} / X)]$, which is true for any sigma-algebra. This is what you called iterated expectation. $\endgroup$ – dada Sep 25 '16 at 20:43
  • $\begingroup$ I still have a question though. I know what is the probability of an event knowing another event, but I've never seen the probability of an event knowing a random variable. So I don't see what $P(g(X) \ne Y | X )$ means ??? $\endgroup$ – dada Sep 26 '16 at 10:06
  • $\begingroup$ $P(g(X) \neq Y)$ is just a number, but $P(g(X) \neq Y \mid X)$ is a random function of $X$. For a more concrete example, imagine you roll a die, call the outcome $X$ and then flip a coin with probability of heads $1 / X$. $P(\text{heads})$ is some fixed number, but $P(\text{heads} \mid X) = 1 / X$. $\endgroup$ – dsaxton Sep 26 '16 at 13:30
  • $\begingroup$ Ok for the intuitive example. But is there a formal definition of the probability knowing a random variable, like that of the conditional expectation ? $\endgroup$ – dada Sep 26 '16 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.