I am trying to understand the theory of this paper. Basically, the paper tries to lay down a framework for using two sample tests using binary classifiers. Let there be two samples $S_p$~$P^n$ and $S_Q$~$Q^m$ So they construct a dataset by pairing the $n$ samples in $S_p$ with a positive label and $m$ samples from $S_Q$ with a negative label. They hypothesize that if the null hypothesis $P=Q$ is true, then the classification accuracy of the binary classifier will remain near chance level. Below the four steps to follow:
Step1: Make the dataset as mentioned above. $D={(x_i,0)_{i=1}^n \cup (y_i,1)_{i=1}^n }= : {(z_i,l_i)}_{i=1}^{2n}$ i.e pair n examples of each class 0 and 1.
Step 2: Randomly shuffle $D$ and split it into disjoint training and testing subsets $D_{tr}$ and $D_{te}$ where $D=D_{tr} \cup D_{te}$ and $n_{te}=|D_{te}|$
Step 3: Train a binary classifier $f:X \to[0,1]$ on $D_{tr}$
Step 4: We assume that $f(z_i)$ is an estimate of the conditional probability distribution $p(l_i=1|z_i)$. The classification accuracy on $D_{te}$ is given by:
$t=1/n_{te} * \Sigma_{(z_i,l_i)} I [ I (f(z_i) > 0.5) = l_i]$
My question lies in this step. What are they trying to do actually? Why is there a $1/2$ in the equation? It seems like they are trying to find the classification accuracy which is the number of corresctly classified examples divided by the total number of examples. Is that true?


Their function $f$ has output range $[0, 1]$, but they're turning that into a hard class prediction with $I(f(z_i) > 0.5)$. It's the same thing as saying: if $f$ outputs 0.4, you want to round it to 0; if it outputs 0.500001, you round that to 1. Then the classification accuracy is the portion of the time that this rounded label, $I(f(z_i) > 0.5)$, is equal to the true label $l_i$.

In this paper, they don't actually do anything with the continuous part of $f$; they could equally well have just used a function $g : X \to \{0, 1\}$ and talk about the accuracy of that, $I(g(z_i) = l_i)$.

  • 1
    $\begingroup$ 1. Thanks a lot for clarifying. So basically I would take equal number of samples from each class, shuffle them and divide them into train and test data. after that train a binary classifier on the train data. Use this trained model on test data to give out probabilities instead of binary values. Now follow the above rule like you said ($I(f(z_i) > 0.5$) as one or else zero. So this way I will get classification accuracy on test data. $\endgroup$ – bandit_king28 Apr 16 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.