0
$\begingroup$

In most examples I find, the various probabilities are just there and it is a simple question of applying Bayes rule to those numbers. However, I'm having a hard time finding details about how to derive/find/discover those probabilities in my use case.

In my case, I have a dataset of support cases - someone who needed help. I have access to the full "population" if you will - all the cases. Once a case is closed, a customer can provide feedback(survey) indicating if they were satisfied or not. Not satisfied would in this case be the "cancer". I want to use Bayes rule to determine the probability that a case will/would get negative feedback or "have cancer". Not every case gets a survey returned, but I have all the surveys and all the cases connected to them.

In my mind, a returned survey is the thing that can be used to definitively label the connected case as negative (cancer) or positive (no cancer), so can be used to help determine the probabilities to use in Bayes rule.

Subsequently, I can use information I have about the case for my "test". For example, did the case take longer than x days to resolve? If it went over x days, that would be a positive test and under x days a negative test result.

So, given I have a dataset of what has happened in the past, how do I create the probabilities to use and apply Bayes rule?

What I've Got

Total Cases: 18942
Total Surveys: 1421 (1297 Positive, 124 negative)

Of 124 negative surveys, 39 were over x days
Of 1297 positive surveys, 111 were over x days

Here is what I've got with my limited understanding:
Prior probability of negative survey (cancer): [Negative Surveys] / [Total Cases] = .0065466

For my test:
Sensitivity (true positive): [Negatives that were > X days] / [All Negatives] = .3145161 False Positive : [Positive surveys > X days] / [All Positives] = .0855821

Now putting that into Bayes Rule, I have:
$P(A) = .0065466$
$P(B | A) = .3145161$
$P(B | ¬A) = .0855821$

Resulting in: (I think...)
$P(A | B) = .023644982$

$\endgroup$
1
$\begingroup$

I don't think you're handling $A$ and $\neg A$ correctly. A positive review is the complement of a negative review only if you've conditioned on a review being left at all--that is, $124/1421$ and $1297/1421$ vs. $124/18942$ and $18818/18942$. Unless we want to get into estimating the probability that someone will complete the survey given their level of satisfaction, I suspect we should drop the total number of cases and focus on just the cases where a survey was created.


This case is convenient because we can directly calculate any of the categories we're interested in. Remember that Bayes comes from the following identity:

$P(A|B)P(B)=P(A,B)=P(B|A)P(A)$

We know that $P(A,B)$, in English "a review is negative and after X days," is $39/1421=0.027$ if we limit ourselves to just reviews.

If we wanted to directly calculate $P(A|B)$, in English "a review is negative given that it was after X days," we would throw away all datapoints without a review after X days and then calculate $P(A)$, which would be $39/150=0.26$. But this is necessarily equal to $P(B|A)P(A)/P(B)$ by the equality above, which is $\frac{39}{124}\frac{124}{1421}\frac{1421}{150}$.

(Working through where each of the numbers in that last line came from may be helpful in seeing how this all must work out to be the same.)


Let's elaborate some on the case where we want to estimate details about the whole population.

Suppose we consider two additional variables, whether or not the user answered a survey $C$ and whether or not they had a bad time $D$. We have no direct visibility into $D|\neg C$, and if we're being totally scrupulous we don't have visibility into $D|A$ either! (Someone may have had a bad experience but given a good review out of charity to the support staff, for example.) But let's simplify the problem and assume that $P(A|C,D)=1$ and $P(\neg A|C,\neg D)=1$, that is reviews are always honest if we receive them, and so instead of $A$ being undefined in the $\neg C$ case we'll redefine $A$ to be "whether the user would have left a bad review if they left a review."

We want to infer something about $P(A)$. We know $P(A|C)$ (which is the same as $P(A)$ above because there I was explicitly conditioning on $C$), and we observe that $P(A,C)=P(C|A)P(A)$. We know the LHS, but we can't directly measure $P(C|A)$. We know $P(A|C)$, but that isn't actually enough because we don't know $P(A|\neg C)$. So all Bayes gives us is a way to turn hypotheses about $P(C|A)$ (how likely a user is to leave a review given their level of satisfaction) into hypotheses about how satisified users are as a whole.

But! We have a third variable, and now we can try to milk it for all it's worth. We might have a hard time believing that users who leave reviews are exactly as satisfied as users who don't (that is, $P(C|A)=P(C|\neg A)$), but we have an easier time believing that's true conditioned on length of service time $x$. That is, $P(C|A,B)=P(C|\neg A,B)$ and $P(C|A,\neg B)=P(C|\neg A,\neg B)$.

I don't think either of those are actually true, but they're a decent starting point. Basically, we can calculate the proportion of good vs. bad reviews for all service time conditions, and then we can do a weighted sum of those proportions by how frequently the entire population was in that service time condition to estimate the level of satisfaction of the population as a whole.)

$\endgroup$
  • $\begingroup$ Ok. This makes sense, but I still feel like I'm not understanding something here. The in classic disease example we somehow know that a certain population were sick and others healthy. In my case I use the survey. Then there is the test, which for me is '> X days', and we know (I assume by using the known population) that the test is not perfect. I'm just trying to wrap my head around that part, because in those cases, the prior P(A) is represented relative to the whole population. $\endgroup$ – Tanner Lindsay Nov 6 '15 at 21:47
  • $\begingroup$ Actually, as I'm thinking more, I'm wondering if this is a sample problem. I'm using ALL the surveys, which have a variety of problems (self-selection bias etc.) So... If I take the survey information that I do have and improve the quality of the sample to make it closer to random, then I will know the overall incidence of "negative" in the larger population right? $\endgroup$ – Tanner Lindsay Nov 6 '15 at 21:49
  • $\begingroup$ So, something that didn't make it into the answer is Jaynes's argument that we should always write probabilities as $P(A|\&)$ rather than $P(A)$, where $\&$ stands for "all of our background knowledge." For any scope we pick for $P(A)$, someone could point out a more general $P(A)$--a negative review left at your institution is a subset of negative interactions at your institution which is a subset of negative support interactions at all institutions, and so on. So the "population" is whatever we want it to be. $\endgroup$ – Matthew Graves Nov 6 '15 at 22:22
  • $\begingroup$ @TannerLindsay I edited the question with some remarks about making inferences about the population. The short version is that Bayes lets you turn assumptions about something that might be easy to estimate (the probability that they'll fill out a survey conditioned on them being happy or unhappy) into estimates on the number of people that are happy or unhappy in the general population. But it doesn't conjure information out of thin air; it only transmutes it. $\endgroup$ – Matthew Graves Nov 6 '15 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.