What's the right t-test for comparing the distribution of a response variable across two categories for the same sample?

Question

I have data where every record has an ID, a numeric value (that represents the number of times each ID took a particular action) and a binary state that represents whether they achieved a goal.

id       action       goal
1              2     FALSE
2             41      TRUE
3             37      TRUE
4              1     FALSE
5              5     FALSE
6              6     FALSE
7             64     FALSE
8              4      TRUE
9             13     FALSE
10             6     FALSE
...           ...    ...

It's approximately 10,000 records. I want to compare if there is a statistically significant difference for the mean number of actions in the group where the goal is true versus the group where the goal is false.

My null hypothesis is there is no difference between the two groups. I'm interested in checking both tails, whether one group is larger or smaller than the other group.

I tried the analysis of variance to test the null hypothesis.

summary(aov(as.numeric(goal) ~ action))

                                   Df Sum Sq Mean Sq F value   Pr(>F)    
action                              1      2  3.9853   16.06 0.000985 ***
Residuals                       10000   2100  0.1795                     
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

My understanding is that this says there is statistically significant evidence to reject the null hypothesis.

I have a few questions. a) is this the right test for a two-tailed test to check the difference of means from the same sample? b) am I reading it correctly? c) what does it mean that they are significantly different? does it mean they come from different populations?

The reason that I ask b is that the distribution looks nearly identical. The mean number of actions when the goal is TRUE is 46.85 and when FALSE is 37.95.

Additionally, the distributions look like they cover right on top of each other.

Answer 1

First of all, Anova (or t-test, since you have only two groups) is appropriate for this kind of research question. However, in your R code to obtain the Anova results, you switched the grouping variable with the dependent variable. The formula should be action ~ goal, instead of the other way around. At the moment, you test whether different values in the number of action have an influence on the mean goal (which doesn't make any sense).

With the number of observations you have, even a small difference in the mean of your two groups will give a significant result. It is your task to interpret whether the difference is also of practical importance. That depends for example on the range of your action variable.

Edit: Actually, when you use a t-test for your data you assume that you dependent variable is normally distributed. Since you have count data, that is not the case. You could look into Poisson / Negative Binomial regression, if you want to choose a test that is truly appropriate for your data.