I am trying to predict the future performance of customers on one of the websites I run. We provide practice tests leading up to a final exam, taken elsewhere. Based on the user's answer to multiple questions (only counting the first time they see the question), I am hoping to predict if they will pass their final exam.
This is part of my dataset for past customer performance
Set A Set B
Q Cor Tot Cor Tot
1 16 34 6 12
3 33 37 11 12
4 41 43 9 11
6 23 31 7 10
7 18 29 5 7
9 32 34 10 10
10 19 33 6 8
11 14 29 5 10
12 34 38 9 10
13 15 25 5 7
Q
is a question identifier, Set A contains the customers who passed the final exam on their first try, Set B contains the customers who did not pass the final exam on their first try.
Cor
indicates how many customers answered that question correctly on their first try. Tot
indicates how many customers have tried answering that question.
In total, set A contains 620 customers, set B contains 162 customers.
Now, assume that our customer Alice whose performance we want to predict answers the questions as follows (1
is correct, 0
is wrong):
Q A
1 1
3 1
4 0
6 1
7 1
9 1
10 0
11 1
12 1
I am assuming this fits well into a naive Bayes classifier:
$p(Pass) = 620 / (162 + 620) \approx 0.7928388747 $ is the probability a random person (who has answered our survey) will pass the test on their first attempt.
$p(Q_n)$ (answering question $n$ correctly) can then be calculated:
$p(Q_1) = \frac{16+6}{34+12} \approx 0.4783$
We need to find $p(Pass|Q_n)$ (probability of passing the exam given answering question $n$ correctly):
$p(Pass|Q_n) = \frac{p(Q_n|Pass) \times p(Pass)}{p(Q_n)}$
For question 1, this is:
$p(Pass|Q_1) \approx \frac{\frac{16}{34} \times 0.7928}{0.4783} \approx 0.7801$
This is where it gets fuzzy for me (not being a mathematician nor statistician, and not having done statistics for quite some years):
I believe I need to get the answer to the following:
$p(Pass|\cap_{i=0}^n{Q_i}) = p(Pass|Q_1 \cap Q_3 \cap \neg{Q_4} \cap Q_6 \cap Q_7 \cap Q_9 \cap \neg{Q_{10}} \cap Q_{11} \cap Q_{12})$
The formula of which I can be expressed as:
$\frac {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)}} {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)} + \frac{\prod_{x=Q_i}{p(\neg Pass|x)}}{p(\neg Pass)}}$
(I am not sure how to express this using math in the formula above, but the idea is to use $p(\neg Q_n) = 1-p(Q_n)$ whenever the user answers incorrectly).
However, when calculating this, the number converges on 1 after just calculating a few terms. None of the customer data I've tried results in a number distinguisable from 1 after only 20 answers to questions have been taken into account.
Am I doing something wrong (my assumption), or is my data actually showing that our logged data cannot predict future performance?