Interpreting false positives of Placebo group with full information

Question

I'd like to make sure that I am interpreting a study correctly; I'm getting tripped up by Bayes Rule. Formulas and data are listed below my question.

Here's the background:

There is a study where users are randomly assigned to a Placebo or Treatment group of a supplement that could improve their health. They also guess whether they are in the Placebo or Treatment group.

The question I want to answer is this:

If a user reports getting better, what is the probability that they wrongly guess they are in the treatment group? (That is, what's the probability that they are are a false positive.)

Mathematically, I was told that it should be represented in a Bayes Rule formula this way:

The data has four states, $T=Treatment$, $\neg{}T=\text{no treatment}$, $B=Better$, $\neg{}B=\text{not better}$, which includes being the same, $C=Correct$, and $\neg{}C=\text{not correct}$.

Bayes theorem for the untreated case where people improved would be $$\pi(\neg{C}|\neg{}T\land{B})=\frac{f(\neg{}T\land{B}|\neg{C})\pi(\neg{C})}{ f(\neg{}T\land{B}|\neg{C})\pi(\neg{C})+ f(\neg{}T\land{B}|{C})\pi({C})}. $$ In this equation, $f$ is your likelihood function and the probability is denoted $\pi$.

So, based on this suggestion, I ran the analysis and broke the proportions down into these groups (whether they guessed correctly, given that they got better, there was no change or no improvement).

Just calculating those who guess correctly

    **This is the proportion of guess correctly** 
    
     guess_status
    correct    0.605355
    wrong      0.394645
    Name: trial_id, dtype: float64
    **This is the proportion outcome and treatment/placebo ** 
    
     condition  improvement_status
    MD         improved              0.076834
               no change             0.257276
               not improved          0.051222
    PL       **improved              0.079162**
               no change             0.223516
               not improved          0.034924
    Name: trial_id, dtype: float64
    **This is the proportion all groups ** 
    
     guess_status  condition  improvement_status
    correct       MD         improved              0.058207
                             no change             0.143190
                             not improved          0.030268
                  PL         improved              0.041909
                             no change             0.142026
                             not improved          0.020955
    wrong         MD         improved              0.018626
                             no change             0.114086
                             not improved          0.020955
                **PL         improved              0.037253**
                             no change             0.081490
                             not improved          0.013970

I think 𝜋(¬𝑇∧𝐵 | ¬𝐶|) = (0.079162 / 0.394645) * 0.394645

I believe 𝜋(¬𝐶|¬𝑇∧𝐵) is ( (0.079162 / 0.394645) * 0.394645) / ( ( (0.079162 / 0.394645) * 0.394645) + ( 0.605355 * ( 0.079162 / 0.605355 ) ) )

the answer is exactly 0.5, which worries me a bit.

Thanks for letting me know if I took the right approach and calculated bayes rule. And, please let me know if I can make this question easier to answer.

Answer 1

First of all, the conditional probability that they wrongly guess they are in the treatment group, conditional on getting better, is:

$$\mathbb{P}(\neg T \land \neg C | B),$$

which is not the same as:

$$\mathbb{P}(\neg C | \neg T \land B).$$

(The latter probability is the conditional probability that they guess their group wrong, conditional on getting better and being in the treatment group.) Now, perhaps you are actually interested in the second probability, in which case you should state its conditioning information correctly. Assuming you are interested in the latter probability, using the values in your table you have:

$$\begin{align} \hat{\mathbb{P}}(\neg C | \neg T \land B) &= \frac{\hat{\mathbb{P}}(\neg C \land \neg T \land B)}{\hat{\mathbb{P}}(\neg T \land B)} \\[6pt] &= \frac{0.037253}{0.037253 + 0.041909} \\[6pt] &= \frac{0.037253}{0.079153} \\[6pt] &= 0.4706455, \\[6pt] \end{align}$$

which is not exactly equal to one-half. Now, getting an answer near to one-half is about what you would expect if you assume that the patient does not have any information on what group they are in (i.e., assuming it is entirely a random guess). So the result of this calculation is not something that would worry me in this analysis.

Of course, you should bear in mind that the values in your table are presumably just sample proportions not the actual probabilities. There are more sophisticated estimators for unknown probabilities than just substituting sample proportions into Bayes rule. If you genuinely want to model this situation using Bayesian analysis you would impose a prior on the probabilities of the outcomes of interest and derive the posterior inference for the unknown probability. Nevertheless, for a quick and nasty analysis, I see nothing alarming in the outcome from the numbers you have here.