So I am a a bit confused here. My test is simple, I have a control population and a treatment population. The treatment population is shown an image before answering a yes/no question and the control population is not shown any image. So my data is something like this:

sample_id date control_or_treatment response
1 12/25/2020 control 1
2 12/25/2020 treatment 0
3 12/25/2020 control 1
4 12/25/2020 control 0
5 12/25/2020 control 1

where each sample_id is unique and response=1 corresponds to a Y and response=0 corresponds to a N response.

I am running a simple independent t test in python to compare their means, something like:


To get the p-value and see if I can reject $H_{0}$. But I thought I could also aggregate my table into something like:

date control_or_treatment y_responses sessions_per_day y_responses_per_session
12/25/2020 control 10 20 0.50
12/25/2020 treatment 7 15 0.47
12/26/2020 control 22 35 0.63
12/26/2020 treatment 18 25 0.72

and then similar to before, run something like this:


And my assumption was that both would be equivalent but it turns out they are not. What information am I losing in the aggregation that renders the results from the daily aggregation different from the raw results? I thought I was comparing the same mean in both: number of Y responses per session.

EDIT: Is this even an appropriate application of hypothesis testing? There is no sample here, I have the entire historical population. What we are trying to establish is, for future customers should we keep showing the image before asking the question or not show the image if the ultimate goal is to get more Y responses.


2 Answers 2


I see no issue with hypothesis testing per se. You claim to have the full population but in fact you do not.

The population is the group you want to make statements about. You intend to make statements about future users, so you haven't sampled all of them.

I do think you are using the wrong test. Although Python will let you use a t-test because you coded your yes/no responses as 1/0, this test requires a.o. that your data is ratio scale and approximately follows a normal distribution. Clearly this is not the case.

For this type of data a Chi-square test combined with a cross-table is more appropriate. If you have more information on the individual users you could also go for logistic regression, but if you're simply interested in the image question a Chi-square test suffices.

  • $\begingroup$ So I should instead be using a chi-square test, thank you, I'll read up on that. In that context, does my question about aggregation matter at all? $\endgroup$
    – doddy
    Dec 26, 2020 at 20:21
  • $\begingroup$ Not in this context. In theory you could do a Chi-square test for each session separately but I don't really see the use of that. It would only serve to increase the problem of multiple testing. The Chi-square test itself requires an aggregated table yes/no response by seen image/not seen image, which is the cross table I mentioned earlier. $\endgroup$ Dec 27, 2020 at 8:40

In your second test, you are giving equal weights to the 4 observations while they actually come from samples with different size. If you don't take into account any effect of the date, you could group them further, with only control vs treatment. The control treatment will then have (0.50x20+0.63x22)/42=0.568, which is not (0.50+0.63)/2=0.565 (the avg response rate considered by your second t-test).

Besides, you are dealing with a categorical outcome so it would be more suitable to do a chisquare test or the Fisher's exact test on the absolute counts of Y/N answers instead of the Y responses per session. That would then also solve your problem of how to account for the amount of responses that day. Apart from that it is a perfectly fine application of hypothesis testing I would say.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.