# Testing difference of means between two samples, without distribution

I have a data set where I am trying to compare a difference of means between two different samples. However, I do not have the standard deviation of the mean for each sample.

The sample data consists of a date, a number of visits, and a total number of times the event of interest. For each visit, the event of interest can be triggered multiple times. Example data below.

+----------+-----------+-------+--------+------------+
|   Date   | Treatment | Vists | Event1 | Avg.Event1 |
+----------+-----------+-------+--------+------------+
| 1/1/2012 | A         |  5392 |  12390 | 2.297      |
| 1/1/2012 | B         |  5489 |  11499 | 2.094      |
| 1/2/2012 | A         |  6030 |  12332 | 2.045      |
| 1/2/2012 | B         |  6200 |  12003 | 1.935      |
+----------+-----------+-------+--------+------------+


So with a data set like that, I assume I can't use hypothesis tests directly to compare the sample means.

Is there a statistical method to test the difference between treatment A and treatment B?

Any suggestions or pointers in the right direction would be greatly appreciated.

Thank you.

• I don't quite understand. What is the mean you are trying to measure - is it the average number of events per visit (over the full time period) for each treatment? Also, I presume you don't have access to the underlying microdata ie visit level, showing for each visit how many events it triggered. Finally, how many dates do you have data for? – Peter Ellis Sep 24 '12 at 20:16
• @PeterEllis Yes, it's the average number of events per visit for each treatment. You are correct that I do not have the underlying microdata / visit level. We generally let our tests run for five to seven days. – TonyH Sep 25 '12 at 12:33

Your data are counts of a sort - each visit results in a count of events. If you're prepared to make your inference conditional on the visits (ie the randomness in how visits happen doesn't matter) and assume that the number of events from each individual visit is an instance of an independent and identically distributed Poisson random variable, then the variance is equal to the mean. You can estimate both variance and mean by pooling the events and visits for each treatment.

Then, the distribution of your estimate for each treatment can be estimated because the average of your 10,000+ Poisson distributed random variables will be normally distributed (because of the central limit theorem), with variance equal to your estimated variance for the original individual variable divided by N (number of visits). This can be used to create a confidence interval for the mean for each treatment and/or a confidence interval for the difference between the two means.

This could also be easily adapted to a hypothesis test if you wanted.

The null hypothesis is that there is no difference between the treatments. Let $X_A$ be the total number of events under treatment A and $N_A$ the number of visits in which those events occur (and similar for $X_B$ and $N_B$). So you are interested in

$d=\frac{X_A}{N_A}-\frac{X_B}{N_B}$

is it significantly different from zero? Because of the assumptions I outlined above, this expression is the difference between two normally distributed independent variables and is itself normally distributed with estimated variance

$\sigma_d^2=\frac{\frac{X_A+X_B}{N_A+N_B}}{N_A}+\frac{\frac{X_A+X_B}{N_A+N_B}}{N_B}$

Under the null hypothesis,

$\frac{d}{\sqrt{\sigma_d^2}}$ is a standard normally distributed variable (ie mean zero, variance 1) and you can compare it to standard tables to see if it is implausibly large (eg for alpha=0.05, if it is >1.96 or <-1.96) in which case you say you have enough evidence to dismiss the null hypothesis.

Disclaimer - the above was written in a bit of haste so check the detail! - but it should be enough to get you started.

• Thank you. I think I understand the direction you are pointing me. So I could calculate a weighted average (pooled mean) using each day, and use that in my hypothesis test. Would you mind being a little more directive in how I would do a hypothesis test using a Poisson distribution? A few links pointing to deeper reading would be perfect. Thank you. – TonyH Sep 25 '12 at 12:47
• @TonyHaenn - I've added some stuff into the answer. – Peter Ellis Sep 26 '12 at 20:15
• @PeterEllis Shouldn't the variance be $\dfrac{(X_A/N_A)^2}{N_A}+\dfrac{(X_B/N_B)^2}{N_B}$? – Boxuan Apr 7 '17 at 18:22