5
$\begingroup$

I have the following table of conditional probabilities.

       | pre work | positive outcome based on pre work type 
 group | done     | multi choice       | symbolic           
-------|----------|--------------------|--------------------
 A     | YES      | 0.51               | 0.33               
 B     | YES      | 0.42               | 0.56               

Looking at the table above, is there any statistical test that I can use to confirm/reject that subjects in groups A who have done pre work did better preparing with symbolic vs multi choice work?

(it looks like the hypothesis is rejected, but the question holds anyway)

$\endgroup$
5
  • $\begingroup$ I really wanted to help but this question seems too long. I think you'll get more attention if you make it as short as possible and erase everything unnecessary. $\endgroup$
    – Yair Daon
    Commented Apr 6, 2017 at 3:53
  • 1
    $\begingroup$ @YairDaon Try now? $\endgroup$
    – Morpheu5
    Commented Apr 7, 2017 at 9:24
  • $\begingroup$ I think that if both groups did prework you cannot draw any conclusion on the effect of prework. Unless you make a prior assumption on those who did not do prework. Also, thanks for making this shorter but it might be too short now. I know I'm being annoying but I'm only trying to help (both by answering your question and by helping you attract more attention). $\endgroup$
    – Yair Daon
    Commented Apr 7, 2017 at 17:01
  • $\begingroup$ Can you clarify which measurements you want to compare? Is it (prework & A & symbolic) versus (prework & A & multi choice)? Or do you want to quantify some sort of effect where the group B treatment simultaneously improves symbolic outcomes and worsens multi choice? $\endgroup$ Commented Apr 8, 2017 at 16:11
  • $\begingroup$ @YairDaon you are right that all did some pre work, but we chose two different types of pre work to reduce the impact of a lack of a true control group. @eric_kernfeld the second. The hypothesis is that symbolic helps more than multi choice. In practice, I don't think we see that effect taking place, so I'm also interested in how I can reject that hypothesis. It seems that the only effect shows between the slice that did and did not do pre work. $\endgroup$
    – Morpheu5
    Commented Apr 10, 2017 at 17:22

1 Answer 1

2
+50
$\begingroup$

You should also know the size of each group to estimate the error of the estimates of your conditional probabilities. If in the A-YES-symbolic group only 1 of 3 people succeeded, the estimate 0.33 of the success rate is less reliable than if it's 1000 of 3000. If you don't have that information, I don't think any statistical test would apply.

Update: ok, you have the information, perfect. The rest is just Z-test for independent proportions. Fill the following table with your data:

       |               |  # of outcomes 
 group | pre work type | positive | total           
-------|---------------|----------|-------
 A     | multi choice  | k_1      | n_1               
       | symbolic      | k_2      | n_2               

Let $p_1$ be the probability of positive outcome in multi choice group, $p_2$ — in symbolic group. To test $H_0\colon p_1=p_2$, calculate $$Z = \frac{\hat{p}_1-\hat{p}_2}{\sqrt{P\left(1-P\right)\left(\frac1{n_1} + \frac1{n_2}\right)}}, $$ $$\hat{p}_1 = \frac{k_1}{n_1}, \;\hat{p}_2=\frac{k_2}{n_2}, \; P = \frac{\hat{p}_1n_1+\hat{p}_2n_2}{n_1+n_2}.$$ Under $H_0$ $Z$ would be distributed as $N\left(0,1\right)$, so, if you have $H_1\colon p_1\neq p_2$, then your p-value is $p=2\left(1-\Phi\left(\left|Z\right|\right)\right)$, where $\Phi$ is a standard normal cumulative distribution function.

It might be a good idea to calculate confidence interval for the difference between probabilities as well, so you would have an estimate of the effect size with explicitly specified uncertainty.

$\endgroup$
4
  • $\begingroup$ In fact I have that information, and that's how I computed the conditional probabilities – they are basically derived with Bayes' formula, as we can assume the events are independent. $\endgroup$
    – Morpheu5
    Commented Apr 10, 2017 at 17:24
  • $\begingroup$ @Morpheu5 As far as I know, usage and Bayes' rule and testing hypotheses don't mix... In a Bayesian setting, all information is encapsulated in the posterior and hypothesis testing is redundant. Anyone cares to refute this statement? I'm interested. $\endgroup$
    – Yair Daon
    Commented Apr 11, 2017 at 1:32
  • $\begingroup$ Evgeniy - great answer, would upvote again if I could. One thing though - the distribution is only approximately normal, is it not? If OP doesn't have too much data this holds in a modified form? Thanks. $\endgroup$
    – Yair Daon
    Commented Apr 12, 2017 at 0:52
  • $\begingroup$ @YairDaon yes, this is CLT-based asymptotical solution. For one-sample problem there is an exact test based on binomial distribution, but I don't think the same exists for two independent samples. $\endgroup$ Commented Apr 12, 2017 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.