Measuring the magnitude and the significance of the difference of two binomial samples I have two samples with binomial distribution. In one of them, the proportion of successes was 10/20 (0.5), and in the other the proportion of successes was 5/20 (0.25). I want to test whether the difference between the two is significant and what is the magnitude of the difference.
I thought of using the following code in R:
prop.test(c(10,5),c(20,20))

Which generates the following result:
2-sample test for equality of proportions with continuity correction

data:  c(10, 5) out of c(20, 20)
X-squared = 1.7067, df = 1, p-value = 0.1914
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.08988258  0.58988258
sample estimates:
prop 1 prop 2 
  0.50   0.25

Question: Can I use the chi square value (1.7067) as a measure of magnitude and the p value (0.1914) as a measure of the significance of the difference between the two samples?
Is there a better way to answer these questions?
 A: Adding to the good answers that talked about statistical significance.  You also asked about magnitude of difference. The simplest way to look at that is to say what you said in your question: One had 50% success, the other had 25% success.
Or, you could look at an odds ratio. You have something like this (formatting tables here is hard, at least for me)
               S      F
A              10    10
B               5    15

So, the OR would be $\frac{10*15}{5*10} = \frac{150}{50} = 3$
And you could use R to get the 95% CI for the OR, if you wanted.
Or you could calculate relative risk: $\frac{10/(10+10)}{5/(5+15)} = \frac{.5}{.25} = 2$  which is as expected.
A: This test takes sample size into account to see if the two
proportions are 'significantly' different in a statistical
sense. Results 10 out of 20 and and 5 out of 20 could easily
occur by chance. As @Henry comments (+1), the same proportions
at larger sample sizes would be significantly different at the 5% level. Double the sample size to consider 20 out of 40 vs. 10 out of 40. Now the P-value is about $0.04 < 0.05 = 5\%.$ [Using R.]
prop.test(c(20,10),c(40,40))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(20, 10) out of c(40, 40)
X-squared = 4.32, df = 1, p-value = 0.03767
alternative hypothesis: two.sided
95 percent confidence interval:
 0.02002206 0.47997794
sample estimates:
prop 1 prop 2 
  0.50   0.25 

Below is a plot of confidence intervals for $n = 16, 20, 24, \dots, 60$ corresponding to $\hat p_1 = 1/4, \hat p_2 = 1/2.$
We see that $n$ must be at least $36$ for the CI for $p_2 - p_1$ not to include $0.$ [For $n < 16$ the normal approximation is unreliable.]

plot(c(-.7,.2),c(11,65), col="white", ylab="n", 
     xlab = "Diff in Proportions") 
 abline(v = 0, col="red")
 abline(h = 36, col="green")
for (i in 4:15) 
 { CI = prop.test(c(i,2*i), c(4*i,4*i))$conf.int 
 lci = CI[1]; uci = CI[2]
 lines(c(lci,uci), c(4*i,4*i), lwd=2, col="blue")
}    

