t-test raw vs date aggregated data

Question

So I am a a bit confused here. My test is simple, I have a control population and a treatment population. The treatment population is shown an image before answering a yes/no question and the control population is not shown any image. So my data is something like this:

sample_id	date	control_or_treatment	response
1	12/25/2020	control	1
2	12/25/2020	treatment	0
3	12/25/2020	control	1
4	12/25/2020	control	0
5	12/25/2020	control	1

where each sample_id is unique and response=1 corresponds to a Y and response=0 corresponds to a N response.

I am running a simple independent t test in python to compare their means, something like:

stats.ttest_ind(treatment_df['response'],control_df['response'])

To get the p-value and see if I can reject $H_{0}$. But I thought I could also aggregate my table into something like:

date	control_or_treatment	y_responses	sessions_per_day	y_responses_per_session
12/25/2020	control	10	20	0.50
12/25/2020	treatment	7	15	0.47
12/26/2020	control	22	35	0.63
12/26/2020	treatment	18	25	0.72

and then similar to before, run something like this:

stats.ttest_ind(treatment_df['y_responses_per_session'],control_df['y_responses_per_session'])

And my assumption was that both would be equivalent but it turns out they are not. What information am I losing in the aggregation that renders the results from the daily aggregation different from the raw results? I thought I was comparing the same mean in both: number of Y responses per session.

EDIT: Is this even an appropriate application of hypothesis testing? There is no sample here, I have the entire historical population. What we are trying to establish is, for future customers should we keep showing the image before asking the question or not show the image if the ultimate goal is to get more Y responses.

Answer 1

I see no issue with hypothesis testing per se. You claim to have the full population but in fact you do not.

The population is the group you want to make statements about. You intend to make statements about future users, so you haven't sampled all of them.

I do think you are using the wrong test. Although Python will let you use a t-test because you coded your yes/no responses as 1/0, this test requires a.o. that your data is ratio scale and approximately follows a normal distribution. Clearly this is not the case.

For this type of data a Chi-square test combined with a cross-table is more appropriate. If you have more information on the individual users you could also go for logistic regression, but if you're simply interested in the image question a Chi-square test suffices.

Answer 2

In your second test, you are giving equal weights to the 4 observations while they actually come from samples with different size. If you don't take into account any effect of the date, you could group them further, with only control vs treatment. The control treatment will then have (0.50x20+0.63x22)/42=0.568, which is not (0.50+0.63)/2=0.565 (the avg response rate considered by your second t-test).

Besides, you are dealing with a categorical outcome so it would be more suitable to do a chisquare test or the Fisher's exact test on the absolute counts of Y/N answers instead of the Y responses per session. That would then also solve your problem of how to account for the amount of responses that day. Apart from that it is a perfectly fine application of hypothesis testing I would say.