Determine probability values for use in Bayes rule

Question

In most examples I find, the various probabilities are just there and it is a simple question of applying Bayes rule to those numbers. However, I'm having a hard time finding details about how to derive/find/discover those probabilities in my use case.

In my case, I have a dataset of support cases - someone who needed help. I have access to the full "population" if you will - all the cases. Once a case is closed, a customer can provide feedback(survey) indicating if they were satisfied or not. Not satisfied would in this case be the "cancer". I want to use Bayes rule to determine the probability that a case will/would get negative feedback or "have cancer". Not every case gets a survey returned, but I have all the surveys and all the cases connected to them.

In my mind, a returned survey is the thing that can be used to definitively label the connected case as negative (cancer) or positive (no cancer), so can be used to help determine the probabilities to use in Bayes rule.

Subsequently, I can use information I have about the case for my "test". For example, did the case take longer than x days to resolve? If it went over x days, that would be a positive test and under x days a negative test result.

So, given I have a dataset of what has happened in the past, how do I create the probabilities to use and apply Bayes rule?

What I've Got

Total Cases: 18942
Total Surveys: 1421 (1297 Positive, 124 negative)

Of 124 negative surveys, 39 were over x days
Of 1297 positive surveys, 111 were over x days

Here is what I've got with my limited understanding:
Prior probability of negative survey (cancer): [Negative Surveys] / [Total Cases] = .0065466

For my test:
Sensitivity (true positive): [Negatives that were > X days] / [All Negatives] = .3145161 False Positive : [Positive surveys > X days] / [All Positives] = .0855821

Now putting that into Bayes Rule, I have:
$P(A) = .0065466$
$P(B | A) = .3145161$
$P(B | ¬A) = .0855821$

Resulting in: (I think...)
$P(A | B) = .023644982$

Answer 1

I don't think you're handling $A$ and $\neg A$ correctly. A positive review is the complement of a negative review only if you've conditioned on a review being left at all--that is, $124/1421$ and $1297/1421$ vs. $124/18942$ and $18818/18942$. Unless we want to get into estimating the probability that someone will complete the survey given their level of satisfaction, I suspect we should drop the total number of cases and focus on just the cases where a survey was created.

---

This case is convenient because we can directly calculate any of the categories we're interested in. Remember that Bayes comes from the following identity:

$P(A|B)P(B)=P(A,B)=P(B|A)P(A)$

We know that $P(A,B)$, in English "a review is negative and after X days," is $39/1421=0.027$ if we limit ourselves to just reviews.

If we wanted to directly calculate $P(A|B)$, in English "a review is negative given that it was after X days," we would throw away all datapoints without a review after X days and then calculate $P(A)$, which would be $39/150=0.26$. But this is necessarily equal to $P(B|A)P(A)/P(B)$ by the equality above, which is $\frac{39}{124}\frac{124}{1421}\frac{1421}{150}$.

(Working through where each of the numbers in that last line came from may be helpful in seeing how this all must work out to be the same.)

---

Let's elaborate some on the case where we want to estimate details about the whole population.

Suppose we consider two additional variables, whether or not the user answered a survey $C$ and whether or not they had a bad time $D$. We have no direct visibility into $D|\neg C$, and if we're being totally scrupulous we don't have visibility into $D|A$ either! (Someone may have had a bad experience but given a good review out of charity to the support staff, for example.) But let's simplify the problem and assume that $P(A|C,D)=1$ and $P(\neg A|C,\neg D)=1$, that is reviews are always honest if we receive them, and so instead of $A$ being undefined in the $\neg C$ case we'll redefine $A$ to be "whether the user would have left a bad review if they left a review."

We want to infer something about $P(A)$. We know $P(A|C)$ (which is the same as $P(A)$ above because there I was explicitly conditioning on $C$), and we observe that $P(A,C)=P(C|A)P(A)$. We know the LHS, but we can't directly measure $P(C|A)$. We know $P(A|C)$, but that isn't actually enough because we don't know $P(A|\neg C)$. So all Bayes gives us is a way to turn hypotheses about $P(C|A)$ (how likely a user is to leave a review given their level of satisfaction) into hypotheses about how satisified users are as a whole.

But! We have a third variable, and now we can try to milk it for all it's worth. We might have a hard time believing that users who leave reviews are exactly as satisfied as users who don't (that is, $P(C|A)=P(C|\neg A)$), but we have an easier time believing that's true conditioned on length of service time $x$. That is, $P(C|A,B)=P(C|\neg A,B)$ and $P(C|A,\neg B)=P(C|\neg A,\neg B)$.

I don't think either of those are actually true, but they're a decent starting point. Basically, we can calculate the proportion of good vs. bad reviews for all service time conditions, and then we can do a weighted sum of those proportions by how frequently the entire population was in that service time condition to estimate the level of satisfaction of the population as a whole.)