# Can I determine the statistical significance of conditional probabilities?

I have the following table of conditional probabilities.

       | pre work | positive outcome based on pre work type
group | done     | multi choice       | symbolic
-------|----------|--------------------|--------------------
A     | YES      | 0.51               | 0.33
B     | YES      | 0.42               | 0.56


Looking at the table above, is there any statistical test that I can use to confirm/reject that subjects in groups A who have done pre work did better preparing with symbolic vs multi choice work?

(it looks like the hypothesis is rejected, but the question holds anyway)

• I really wanted to help but this question seems too long. I think you'll get more attention if you make it as short as possible and erase everything unnecessary. Commented Apr 6, 2017 at 3:53
• @YairDaon Try now? Commented Apr 7, 2017 at 9:24
• I think that if both groups did prework you cannot draw any conclusion on the effect of prework. Unless you make a prior assumption on those who did not do prework. Also, thanks for making this shorter but it might be too short now. I know I'm being annoying but I'm only trying to help (both by answering your question and by helping you attract more attention). Commented Apr 7, 2017 at 17:01
• Can you clarify which measurements you want to compare? Is it (prework & A & symbolic) versus (prework & A & multi choice)? Or do you want to quantify some sort of effect where the group B treatment simultaneously improves symbolic outcomes and worsens multi choice? Commented Apr 8, 2017 at 16:11
• @YairDaon you are right that all did some pre work, but we chose two different types of pre work to reduce the impact of a lack of a true control group. @eric_kernfeld the second. The hypothesis is that symbolic helps more than multi choice. In practice, I don't think we see that effect taking place, so I'm also interested in how I can reject that hypothesis. It seems that the only effect shows between the slice that did and did not do pre work. Commented Apr 10, 2017 at 17:22

You should also know the size of each group to estimate the error of the estimates of your conditional probabilities. If in the A-YES-symbolic group only 1 of 3 people succeeded, the estimate 0.33 of the success rate is less reliable than if it's 1000 of 3000. If you don't have that information, I don't think any statistical test would apply.

Update: ok, you have the information, perfect. The rest is just Z-test for independent proportions. Fill the following table with your data:

       |               |  # of outcomes
group | pre work type | positive | total
-------|---------------|----------|-------
A     | multi choice  | k_1      | n_1
| symbolic      | k_2      | n_2


Let $p_1$ be the probability of positive outcome in multi choice group, $p_2$ — in symbolic group. To test $H_0\colon p_1=p_2$, calculate $$Z = \frac{\hat{p}_1-\hat{p}_2}{\sqrt{P\left(1-P\right)\left(\frac1{n_1} + \frac1{n_2}\right)}},$$ $$\hat{p}_1 = \frac{k_1}{n_1}, \;\hat{p}_2=\frac{k_2}{n_2}, \; P = \frac{\hat{p}_1n_1+\hat{p}_2n_2}{n_1+n_2}.$$ Under $H_0$ $Z$ would be distributed as $N\left(0,1\right)$, so, if you have $H_1\colon p_1\neq p_2$, then your p-value is $p=2\left(1-\Phi\left(\left|Z\right|\right)\right)$, where $\Phi$ is a standard normal cumulative distribution function.

It might be a good idea to calculate confidence interval for the difference between probabilities as well, so you would have an estimate of the effect size with explicitly specified uncertainty.

• In fact I have that information, and that's how I computed the conditional probabilities – they are basically derived with Bayes' formula, as we can assume the events are independent. Commented Apr 10, 2017 at 17:24
• @Morpheu5 As far as I know, usage and Bayes' rule and testing hypotheses don't mix... In a Bayesian setting, all information is encapsulated in the posterior and hypothesis testing is redundant. Anyone cares to refute this statement? I'm interested. Commented Apr 11, 2017 at 1:32
• Evgeniy - great answer, would upvote again if I could. One thing though - the distribution is only approximately normal, is it not? If OP doesn't have too much data this holds in a modified form? Thanks. Commented Apr 12, 2017 at 0:52
• @YairDaon yes, this is CLT-based asymptotical solution. For one-sample problem there is an exact test based on binomial distribution, but I don't think the same exists for two independent samples. Commented Apr 12, 2017 at 12:09