1
$\begingroup$

I am trying to predict the future performance of customers on one of the websites I run. We provide practice tests leading up to a final exam, taken elsewhere. Based on the user's answer to multiple questions (only counting the first time they see the question), I am hoping to predict if they will pass their final exam.

This is part of my dataset for past customer performance

    Set A       Set B
Q   Cor Tot Cor Tot
1   16  34  6   12
3   33  37  11  12
4   41  43  9   11
6   23  31  7   10
7   18  29  5   7
9   32  34  10  10
10  19  33  6   8
11  14  29  5   10
12  34  38  9   10
13  15  25  5   7

Q is a question identifier, Set A contains the customers who passed the final exam on their first try, Set B contains the customers who did not pass the final exam on their first try.

Cor indicates how many customers answered that question correctly on their first try. Tot indicates how many customers have tried answering that question.

In total, set A contains 620 customers, set B contains 162 customers.

Now, assume that our customer Alice whose performance we want to predict answers the questions as follows (1 is correct, 0 is wrong):

Q   A
1   1
3   1
4   0
6   1
7   1
9   1
10  0
11  1
12  1

I am assuming this fits well into a naive Bayes classifier:

$p(Pass) = 620 / (162 + 620) \approx 0.7928388747 $ is the probability a random person (who has answered our survey) will pass the test on their first attempt.

$p(Q_n)$ (answering question $n$ correctly) can then be calculated:

$p(Q_1) = \frac{16+6}{34+12} \approx 0.4783$

We need to find $p(Pass|Q_n)$ (probability of passing the exam given answering question $n$ correctly):

$p(Pass|Q_n) = \frac{p(Q_n|Pass) \times p(Pass)}{p(Q_n)}$

For question 1, this is:

$p(Pass|Q_1) \approx \frac{\frac{16}{34} \times 0.7928}{0.4783} \approx 0.7801$

This is where it gets fuzzy for me (not being a mathematician nor statistician, and not having done statistics for quite some years):

I believe I need to get the answer to the following:

$p(Pass|\cap_{i=0}^n{Q_i}) = p(Pass|Q_1 \cap Q_3 \cap \neg{Q_4} \cap Q_6 \cap Q_7 \cap Q_9 \cap \neg{Q_{10}} \cap Q_{11} \cap Q_{12})$

The formula of which I can be expressed as:

$\frac {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)}} {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)} + \frac{\prod_{x=Q_i}{p(\neg Pass|x)}}{p(\neg Pass)}}$

(I am not sure how to express this using math in the formula above, but the idea is to use $p(\neg Q_n) = 1-p(Q_n)$ whenever the user answers incorrectly).

However, when calculating this, the number converges on 1 after just calculating a few terms. None of the customer data I've tried results in a number distinguisable from 1 after only 20 answers to questions have been taken into account.

Am I doing something wrong (my assumption), or is my data actually showing that our logged data cannot predict future performance?

Improve this question
$\endgroup$
2
  • $\begingroup$ There's actually a third piece of information/ probability for each question: if it was not answered on their first try. That might have more predictive value than everything else, given that the pass and fail populations have seem to have very similar correctness probabilities (as @Henry said). But we don't know if not answering on your first attempt is penalized, and your dataset doesn't say whether the user can go back and reattempt it later (out of order? 2nd complete run through questions? nth run through questions?) $\endgroup$ – smci May 30 '15 at 0:42
  • $\begingroup$ You're also not telling us whether later questions score more, so whether skipping early questions is a good or bad tactic by the candidates. $\endgroup$ – smci May 30 '15 at 0:44
3
$\begingroup$

I don't think you have much hope of predicting well from this: for most of the your questions (apart from Q4 and Q6), a smaller proportion of those who passed the actual exam correctly answered your questions than those who failed the actual exam.

For example I would say your data suggests $\Pr(\text{Pass}|Q_1 \text{ correct}) = \frac{16}{16+6} \approx 0.72$ while $\Pr(\text{Pass}|Q_1 \text{ incorrect}) = \frac{34-16}{34-16+12-6} \approx 0.75$, which is slightly bigger. Incidentally, it looks as if $\Pr(\text{Pass}|Q_1 \text{ not answered}) = \frac{586}{736} \approx 0.80$.

So I suspect your logged data cannot predict future performance.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ That is unfortunately just a byproduct of the cut-down dataset I provided, for the full list of questions (around 500 or so) there is a marked difference for certain questions in the other direction. I do see your point, however, but can this be remedied by picking which questions I use based on how large the difference is? I can easily pick 20-30 questions to use that have a much larger proportion of customers who passed the exam than failed. I thought that using the full dataset (although not in this short example) would be beneficial to the end result. $\endgroup$ – Vegard Larsen Aug 29 '11 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.