Naive Bayes classifier for predicting performance

Question

I am trying to predict the future performance of customers on one of the websites I run. We provide practice tests leading up to a final exam, taken elsewhere. Based on the user's answer to multiple questions (only counting the first time they see the question), I am hoping to predict if they will pass their final exam.

This is part of my dataset for past customer performance

    Set A       Set B
Q   Cor Tot Cor Tot
1   16  34  6   12
3   33  37  11  12
4   41  43  9   11
6   23  31  7   10
7   18  29  5   7
9   32  34  10  10
10  19  33  6   8
11  14  29  5   10
12  34  38  9   10
13  15  25  5   7

Q is a question identifier, Set A contains the customers who passed the final exam on their first try, Set B contains the customers who did not pass the final exam on their first try.

Cor indicates how many customers answered that question correctly on their first try. Tot indicates how many customers have tried answering that question.

In total, set A contains 620 customers, set B contains 162 customers.

---

Now, assume that our customer Alice whose performance we want to predict answers the questions as follows (1 is correct, 0 is wrong):

Q   A
1   1
3   1
4   0
6   1
7   1
9   1
10  0
11  1
12  1

---

I am assuming this fits well into a naive Bayes classifier:

$p(Pass) = 620 / (162 + 620) \approx 0.7928388747 $ is the probability a random person (who has answered our survey) will pass the test on their first attempt.

$p(Q_n)$ (answering question $n$ correctly) can then be calculated:

$p(Q_1) = \frac{16+6}{34+12} \approx 0.4783$

We need to find $p(Pass|Q_n)$ (probability of passing the exam given answering question $n$ correctly):

$p(Pass|Q_n) = \frac{p(Q_n|Pass) \times p(Pass)}{p(Q_n)}$

For question 1, this is:

$p(Pass|Q_1) \approx \frac{\frac{16}{34} \times 0.7928}{0.4783} \approx 0.7801$

This is where it gets fuzzy for me (not being a mathematician nor statistician, and not having done statistics for quite some years):

I believe I need to get the answer to the following:

$p(Pass|\cap_{i=0}^n{Q_i}) = p(Pass|Q_1 \cap Q_3 \cap \neg{Q_4} \cap Q_6 \cap Q_7 \cap Q_9 \cap \neg{Q_{10}} \cap Q_{11} \cap Q_{12})$

The formula of which I can be expressed as:

$\frac {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)}} {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)} + \frac{\prod_{x=Q_i}{p(\neg Pass|x)}}{p(\neg Pass)}}$

(I am not sure how to express this using math in the formula above, but the idea is to use $p(\neg Q_n) = 1-p(Q_n)$ whenever the user answers incorrectly).

However, when calculating this, the number converges on 1 after just calculating a few terms. None of the customer data I've tried results in a number distinguisable from 1 after only 20 answers to questions have been taken into account.

Am I doing something wrong (my assumption), or is my data actually showing that our logged data cannot predict future performance?

Answer 1

I don't think you have much hope of predicting well from this: for most of the your questions (apart from Q4 and Q6), a smaller proportion of those who passed the actual exam correctly answered your questions than those who failed the actual exam.

For example I would say your data suggests $\Pr(\text{Pass}|Q_1 \text{ correct}) = \frac{16}{16+6} \approx 0.72$ while $\Pr(\text{Pass}|Q_1 \text{ incorrect}) = \frac{34-16}{34-16+12-6} \approx 0.75$, which is slightly bigger. Incidentally, it looks as if $\Pr(\text{Pass}|Q_1 \text{ not answered}) = \frac{586}{736} \approx 0.80$.

So I suspect your logged data cannot predict future performance.