# Naive Bayes classifier for predicting performance

I am trying to predict the future performance of customers on one of the websites I run. We provide practice tests leading up to a final exam, taken elsewhere. Based on the user's answer to multiple questions (only counting the first time they see the question), I am hoping to predict if they will pass their final exam.

This is part of my dataset for past customer performance

    Set A       Set B
Q   Cor Tot Cor Tot
1   16  34  6   12
3   33  37  11  12
4   41  43  9   11
6   23  31  7   10
7   18  29  5   7
9   32  34  10  10
10  19  33  6   8
11  14  29  5   10
12  34  38  9   10
13  15  25  5   7


Q is a question identifier, Set A contains the customers who passed the final exam on their first try, Set B contains the customers who did not pass the final exam on their first try.

Cor indicates how many customers answered that question correctly on their first try. Tot indicates how many customers have tried answering that question.

In total, set A contains 620 customers, set B contains 162 customers.

Now, assume that our customer Alice whose performance we want to predict answers the questions as follows (1 is correct, 0 is wrong):

Q   A
1   1
3   1
4   0
6   1
7   1
9   1
10  0
11  1
12  1


I am assuming this fits well into a naive Bayes classifier:

$p(Pass) = 620 / (162 + 620) \approx 0.7928388747$ is the probability a random person (who has answered our survey) will pass the test on their first attempt.

$p(Q_n)$ (answering question $n$ correctly) can then be calculated:

$p(Q_1) = \frac{16+6}{34+12} \approx 0.4783$

We need to find $p(Pass|Q_n)$ (probability of passing the exam given answering question $n$ correctly):

$p(Pass|Q_n) = \frac{p(Q_n|Pass) \times p(Pass)}{p(Q_n)}$

For question 1, this is:

$p(Pass|Q_1) \approx \frac{\frac{16}{34} \times 0.7928}{0.4783} \approx 0.7801$

This is where it gets fuzzy for me (not being a mathematician nor statistician, and not having done statistics for quite some years):

I believe I need to get the answer to the following:

$p(Pass|\cap_{i=0}^n{Q_i}) = p(Pass|Q_1 \cap Q_3 \cap \neg{Q_4} \cap Q_6 \cap Q_7 \cap Q_9 \cap \neg{Q_{10}} \cap Q_{11} \cap Q_{12})$

The formula of which I can be expressed as:

$\frac {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)}} {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)} + \frac{\prod_{x=Q_i}{p(\neg Pass|x)}}{p(\neg Pass)}}$

(I am not sure how to express this using math in the formula above, but the idea is to use $p(\neg Q_n) = 1-p(Q_n)$ whenever the user answers incorrectly).

However, when calculating this, the number converges on 1 after just calculating a few terms. None of the customer data I've tried results in a number distinguisable from 1 after only 20 answers to questions have been taken into account.

Am I doing something wrong (my assumption), or is my data actually showing that our logged data cannot predict future performance?

• There's actually a third piece of information/ probability for each question: if it was not answered on their first try. That might have more predictive value than everything else, given that the pass and fail populations have seem to have very similar correctness probabilities (as @Henry said). But we don't know if not answering on your first attempt is penalized, and your dataset doesn't say whether the user can go back and reattempt it later (out of order? 2nd complete run through questions? nth run through questions?)
– smci
May 30 '15 at 0:42
• You're also not telling us whether later questions score more, so whether skipping early questions is a good or bad tactic by the candidates.
– smci
May 30 '15 at 0:44

For example I would say your data suggests $\Pr(\text{Pass}|Q_1 \text{ correct}) = \frac{16}{16+6} \approx 0.72$ while $\Pr(\text{Pass}|Q_1 \text{ incorrect}) = \frac{34-16}{34-16+12-6} \approx 0.75$, which is slightly bigger. Incidentally, it looks as if $\Pr(\text{Pass}|Q_1 \text{ not answered}) = \frac{586}{736} \approx 0.80$.