Determine if three is statistically different than ten for a very large number of observations (1,000,000) For 1,000,000 observations, I observed a discrete event, X, 3 times for the control group and 10 times for the test group. How do I determine for a large number of observations (1,000,000), if three is statistically different than ten?
 A: A (two-sided) Fisher's Exact test gives p-value = 0.092284.
function p = fexact(k, x, m, n)
%FEXACT Fisher's Exact test.
%   Y = FEXACT(K, X, M, N) calculates the P-value for Fisher's
%   Exact Test.
%   K, X, M and N must be nonnegative integer vectors of the same
%   length.  The following must also hold:
%   X <= N <= M, X <= K <= M and K + N - M <= X.  Here:
%   K is the number of items in the group,
%   X is the number of items in the group with the feature,
%   M is the total number of items,
%   N is the total number of items with the feature,

if nargin < 4
   help(mfilename);
   return;
end
nr = length(k);
if nr ~= length(x) | nr ~= length(m) | nr ~= length(n)
   help(mfilename);
   return;
end

na = nan;
v = na(ones(nr, 1));
mi = max(0, k + n - m);
ma = min(k, n);

d = hygepdf(x, m, k, n) * (1 + 5.8e-11);
for i = 1:nr
  y = hygepdf(mi(i):ma(i), m(i), k(i), n(i));
  v(i) = sum(y(y <= d(i)));
end
p = max(min(v, 1), 0);
p(isnan(v)) = nan;

For your example, try fexact(1e6, 3, 2e6, 13).
A: I think a simple chi-squared test will do the trick. Do you have 1,000,000 observations for both control and test? If so, your table of observations will be (in R code)
Edit: Woops! Left off a zero!
m <- rbind(c(3, 1000000-3), c(10, 1000000-10))
#      [,1]   [,2] 
# [1,]    3 999997
# [2,]   10 999990

And chi-squared test will be
chisq.test(m)

Which returns chi-squared = 2.7692, df = 1, p-value = 0.0961, which is not statistically significant at the p < 0.05 level. I'd be surprised if these could be clinically significant anyway.
A: In this case Poisson is good approximation for distribution for number of cases.
There is simple formula to approximate variance of log RR (delta method) .
log RR = 10/3 = 1.2, 
se log RR = sqrt(1/3+1/10) = 0.66, so 95%CI = (-0.09; 2.5)
It is not significant difference at 0.05 level using two-sided test.
LR based Chi-square test for Poisson model gives p=0.046 and Wald test p=0.067.
This results are similar to Pearson Chi-square test without continuity correction (Chi2 with correction p=0.096).
Another possibility is chisq.test with option simulate.p.value=T, in this case p=0.092 (for 100 000 simulations).
In this case test statistics is rather discrete, so Fisher test can be conservative.
There is some evidence that difference can be significant. Before final conclusion data collecting process should be taken into account.
A: The huge denominators throw off one's intuition. Since the sample sizes are identical, and the proportions low, the problem can be recast: 13 events occurred, and were expected (by null hypothesis) to occur equally in both groups. In fact the split was 3 in one group and 10 in the other. How rare is that? The binomial test answers.
Enter this line into R:
binom.test(3,13,0.5,alternative="two.sided")
The two-tail P value is 0.09229, identical to four digits to the results of Fisher's test. 
Looked at that way, the results are not surprising. The problem is equivalent to this one: If you flipped a coin 13 times, how surprising would it be to see three or fewer, or ten or more, heads. One of those outcomes would occur 9.23% of the time. 
A: I would be really surprised if you find the difference statistically significant. Having said that you may want to use a test for a difference of proportions (3 out of 1M vs 10 out of 1M).
A: In addition to the other answers:
If you have 1,000,000 observations and when your event comes up only a few times, you are likely to want to look at a lot of different events.
If you look at 100 different events you will run into problems if you work with p<0.05 as criteria for significance. 
