Comparing two users' average number of events per day with a statistical approach

Question

I've recently had a test which included a question that was phrased like this:

Compare user A and B; would you conclude their average number of events per day to be different

The dataset was over a period of a month. It was a relational database where every event a was associated with a user and sent at a datetime. I'll give an example of what the dataset looked like

user_id,time_of_event,event_name,event_parameters
A,2018-03-01 17:21:44, sign_up, '{"source": "mobile_app"}'
A,2018-03-01 17:21:54, start_tutorial, '{"experiment_group": "control"}'
B,2018-03-02 05:33:17, session_start, '{"session_medium": "webapp"}'
B,2018-03-02 05:36:35, add_to_cart, '{"item_id": 132156, "price": 12.99}'
...

I thought it was a pretty simple question, and my approach was pretty simple.

1. Get the average number of events that user A over the month, adding in 0's for days on which they sent no events.
2. Repeat 1 but for B.
3. Compare the averages.

When I was given feedback on this test, I was told that this response was unsatisfactory because it lacked a "statistical approach" to coming to the answer.

What statistical approach could I have used to come to a higher quality answer?

Answer 1

What was probably meant by the feedback was that you didn't consider the possibility that, because of stochastic variations, the averages you computed could be deceiving. Even if e.g. the average number for A was larger than the average number for B, the real rate for A could still be smaller than that of B, just by chance. So what they were probably looking for was some kind of significance test.

The problem is that it is not really possible to properly apply such a test if we don't know more about how the data was generated. The way your test question is phrased might suggest that they expect you to presume a Poisson process. But from the description of the events in your data set excerpt, I could imagine that e.g. independence of the events is violated, which would, however, be required for the Poisson process.

Nevertheless, presuming that your test was not meant to be too difficult, I proceed assuming that the process is, indeed, a Poisson process. Thus we presume that the count of the events follows in both cases a Poisson distribution. The question now is whether there is significant evidence that the event rates are different. Fortunately, there exists for this situation an exact significance test, sometimes called the Poisson test, and the stats package in R even conveniently provides an implementation.

Answer 2

I think a stronger answer would have been to recommend using a t-test to compare the distribution of each users' events during a given month.

A t-test takes the analysis you did one step further by applying a hypothesis test to it. One reason we typically don't rely on comparisons of the means alone is that two groups (in this case, each users' respective event count), may appear slightly different, but actually be the same.

Let's consider another example. Let's say we want to understand if chipmunks are heavier, on average, than hamsters. We take 20 chipmunks and 20 hamsters and weigh them. Now, let's say that the chipmunks' average weight is 10 ounces (I have absolutely no clue if this is a reasonable assumption!) and the hamsters' average weight was 10.5 ounces. We might just declare hamsters the winner, but we'd be forgetting one important question - "is the difference a true difference, or is it reasonably the result of chance". The difference in their weights is subtle. Can we really assume that it wasn't a mere chance they were different? A t-test allows us to assign a level of confidence to our analysis.

In your example, a t-test would allow you to set a confidence level to determine if one user's total event count was statistically significantly larger than the other.