Testing difference of means between two samples, without distribution

Question

I have a data set where I am trying to compare a difference of means between two different samples. However, I do not have the standard deviation of the mean for each sample.

The sample data consists of a date, a number of visits, and a total number of times the event of interest. For each visit, the event of interest can be triggered multiple times. Example data below.

+----------+-----------+-------+--------+------------+
|   Date   | Treatment | Vists | Event1 | Avg.Event1 |
+----------+-----------+-------+--------+------------+
| 1/1/2012 | A         |  5392 |  12390 | 2.297      |
| 1/1/2012 | B         |  5489 |  11499 | 2.094      |
| 1/2/2012 | A         |  6030 |  12332 | 2.045      |
| 1/2/2012 | B         |  6200 |  12003 | 1.935      |
+----------+-----------+-------+--------+------------+

So with a data set like that, I assume I can't use hypothesis tests directly to compare the sample means.

Is there a statistical method to test the difference between treatment A and treatment B?

Any suggestions or pointers in the right direction would be greatly appreciated.

Thank you.

Answer 1

Your data are counts of a sort - each visit results in a count of events. If you're prepared to make your inference conditional on the visits (ie the randomness in how visits happen doesn't matter) and assume that the number of events from each individual visit is an instance of an independent and identically distributed Poisson random variable, then the variance is equal to the mean. You can estimate both variance and mean by pooling the events and visits for each treatment.

Then, the distribution of your estimate for each treatment can be estimated because the average of your 10,000+ Poisson distributed random variables will be normally distributed (because of the central limit theorem), with variance equal to your estimated variance for the original individual variable divided by N (number of visits). This can be used to create a confidence interval for the mean for each treatment and/or a confidence interval for the difference between the two means.

This could also be easily adapted to a hypothesis test if you wanted.

Edit addition - hypothesis test

The null hypothesis is that there is no difference between the treatments. Let $X_A$ be the total number of events under treatment A and $N_A$ the number of visits in which those events occur (and similar for $X_B$ and $N_B$). So you are interested in

$d=\frac{X_A}{N_A}-\frac{X_B}{N_B}$

is it significantly different from zero? Because of the assumptions I outlined above, this expression is the difference between two normally distributed independent variables and is itself normally distributed with estimated variance

$\sigma_d^2=\frac{\frac{X_A+X_B}{N_A+N_B}}{N_A}+\frac{\frac{X_A+X_B}{N_A+N_B}}{N_B}$

Under the null hypothesis,

$\frac{d}{\sqrt{\sigma_d^2}}$ is a standard normally distributed variable (ie mean zero, variance 1) and you can compare it to standard tables to see if it is implausibly large (eg for alpha=0.05, if it is >1.96 or <-1.96) in which case you say you have enough evidence to dismiss the null hypothesis.

Disclaimer - the above was written in a bit of haste so check the detail! - but it should be enough to get you started.