1
$\begingroup$

I've recently had a test which included a question that was phrased like this:

Compare user A and B; would you conclude their average number of events per day to be different

The dataset was over a period of a month. It was a relational database where every event a was associated with a user and sent at a datetime. I'll give an example of what the dataset looked like

user_id,time_of_event,event_name,event_parameters
A,2018-03-01 17:21:44, sign_up, '{"source": "mobile_app"}'
A,2018-03-01 17:21:54, start_tutorial, '{"experiment_group": "control"}'
B,2018-03-02 05:33:17, session_start, '{"session_medium": "webapp"}'
B,2018-03-02 05:36:35, add_to_cart, '{"item_id": 132156, "price": 12.99}'
...

I thought it was a pretty simple question, and my approach was pretty simple.

  1. Get the average number of events that user A over the month, adding in 0's for days on which they sent no events.
  2. Repeat 1 but for B.
  3. Compare the averages.

When I was given feedback on this test, I was told that this response was unsatisfactory because it lacked a "statistical approach" to coming to the answer.

What statistical approach could I have used to come to a higher quality answer?

$\endgroup$

2 Answers 2

1
+50
$\begingroup$

What was probably meant by the feedback was that you didn't consider the possibility that, because of stochastic variations, the averages you computed could be deceiving. Even if e.g. the average number for A was larger than the average number for B, the real rate for A could still be smaller than that of B, just by chance. So what they were probably looking for was some kind of significance test.

The problem is that it is not really possible to properly apply such a test if we don't know more about how the data was generated. The way your test question is phrased might suggest that they expect you to presume a Poisson process. But from the description of the events in your data set excerpt, I could imagine that e.g. independence of the events is violated, which would, however, be required for the Poisson process.

Nevertheless, presuming that your test was not meant to be too difficult, I proceed assuming that the process is, indeed, a Poisson process. Thus we presume that the count of the events follows in both cases a Poisson distribution. The question now is whether there is significant evidence that the event rates are different. Fortunately, there exists for this situation an exact significance test, sometimes called the Poisson test, and the stats package in R even conveniently provides an implementation.

$\endgroup$
9
  • $\begingroup$ If the sample is large then a t-test might work well. Without knowing the underlying distribution, one can still assume that the sample mean (which is what's being compared) will follow approximately a normal distribution, and the estimate of the variance will be approximately chi-squared distributed. $\endgroup$ Oct 2 at 15:47
  • $\begingroup$ @SextusEmpiricus Sure. I did not mean to claim that the t-test would not be a good approximation. $\endgroup$
    – frank
    Oct 2 at 15:54
  • $\begingroup$ I was mostly referring to the sentence "The problem is that it is not really possible to properly apply such a test if we don't know more about how the data was generated." $\endgroup$ Oct 2 at 16:12
  • $\begingroup$ Also, using the assumption of a Poisson distribution might not be a good idea. The distribution can be over-dispersed, either because events are clustered or because the average event rates are not constant (and possibly zero-inflated as well). If one would base an estimate for the variation on the observed number of cases then one would underestimate the variation. $\endgroup$ Oct 2 at 16:13
  • $\begingroup$ @SextusEmpiricus I stated that I have my doubts about Poisson, too. And t-test, too, should not be applied if we don't know more about the data generation process. $\endgroup$
    – frank
    Oct 2 at 16:17
1
$\begingroup$

I think a stronger answer would have been to recommend using a t-test to compare the distribution of each users' events during a given month.

A t-test takes the analysis you did one step further by applying a hypothesis test to it. One reason we typically don't rely on comparisons of the means alone is that two groups (in this case, each users' respective event count), may appear slightly different, but actually be the same.

Let's consider another example. Let's say we want to understand if chipmunks are heavier, on average, than hamsters. We take 20 chipmunks and 20 hamsters and weigh them. Now, let's say that the chipmunks' average weight is 10 ounces (I have absolutely no clue if this is a reasonable assumption!) and the hamsters' average weight was 10.5 ounces. We might just declare hamsters the winner, but we'd be forgetting one important question - "is the difference a true difference, or is it reasonably the result of chance". The difference in their weights is subtle. Can we really assume that it wasn't a mere chance they were different? A t-test allows us to assign a level of confidence to our analysis.

In your example, a t-test would allow you to set a confidence level to determine if one user's total event count was statistically significantly larger than the other.

$\endgroup$
1
  • $\begingroup$ Thanks for the answer, I didn't think about t-tests and took the opportunity to refresh my memory. The 1st assumption for a t-test is that the underlying data is continuous or ordinal. That is the case with your chipmunk and hamster example, but it is not the case with event data. Events are discrete data (there is a finite set, they cannot be divided, it can be counted, etc.). $\endgroup$
    – Cold Fish
    Sep 21 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.