5
$\begingroup$

In marketing, people assume that individual customer purchases are distributed according to a poisson distribution. This may be an incorrect assumption, however all models are wrong but some are useful.

I've managed to find some customer transaction data housing that contains the customer ID and date of purchase as rows for several thousand customers (available here). Here is a sample of the data:

CustomerID  InvoiceDate
17381.0 2011-10-04 08:30:00
15749.0 2011-04-18 13:08:00
14371.0 2011-02-17 16:35:00
18226.0 2011-07-03 10:47:00
14012.0 2011-03-07 12:26:00
13144.0 2011-01-11 11:15:00
12852.0 2011-02-18 08:47:00
12901.0 2011-06-23 16:00:00
13854.0 2011-03-21 11:20:00
13158.0 2011-01-25 09:58:00

The data is publicly available, so I don't think there are any issues posting it here.

I can group the customers and count their transactions for any period of time. I'd like to check how well this assumption of being poisson distributed is.

Could I just count the transactions for a given customer weekly, and then use the chi-square goodness of fit test to determine if the data could have been distributed according to a poisson distribution? Am I missing something crucial in my assumptions?

$\endgroup$
  • $\begingroup$ Start with checking for overdispersion. $\endgroup$ – Aksakal Apr 13 '17 at 19:00
  • $\begingroup$ The example is helpful. It appears the distribution in time will not be uniform: late evening and early morning hours are not represented. That alone will rule out a Poisson process. (Even when the distribution is uniform, the process might not be Poisson: you would then proceed to check for independence.) Your problem is essentially the same as testing a random number generator. Indeed, you can reduce it to that problem by computing the time differences, taking the reciprocals of their exponentials, and testing whether that is an iid sequence of uniform$(0,1)$ values. $\endgroup$ – whuber Apr 13 '17 at 19:16
  • $\begingroup$ Coincidentally, I just visited stats.stackexchange.com to pose a very similar question -- and this guy beat me to it by a couple of hours. Is that a Poisson process? ;) However, I would probably have asked a slightly different question since my data involve a lot of sparse binary events with occasional counts > 1 due to time resolution. If he had removed the time of day, the question might become similar to mine. $\endgroup$ – James Bowery Apr 13 '17 at 19:31
  • $\begingroup$ @whuber what if we removed the time of day and worried only about the day? What I'm most interested is the distribution of sales for a given individual. So if I take customer x, and examine the number of sales per week for customer x, is that poisson distributed? $\endgroup$ – Demetri Pananos Apr 13 '17 at 20:57
  • $\begingroup$ Even if it's not Poisson (according to some tests) it doesn't mean that you shouldn't model it as Poisson, by the way $\endgroup$ – Aksakal Apr 14 '17 at 0:00
1
$\begingroup$

If you have data from a Poisson process with parameter $\lambda$, then the inter-arrival times $$ X_1, \ldots, X_n \overset{iid}{\sim} \text{Exponential}(\lambda). $$ You get these data points by just taking the difference of the column of time stamps. Then you can just look at a qqplot (check this thread for R code.)

Another idea: bin the data into buckets. Then run a Poisson regression on a few polynomials in time. If any of the significance tests reject the null, you can't assume that it's a (homogeneous) Poisson process.

$\endgroup$
  • $\begingroup$ This checks uniformity, but it says nothing about independence, which is an important property of a Poisson process. $\endgroup$ – whuber Apr 13 '17 at 23:53
  • $\begingroup$ @whuber that's true $\endgroup$ – Taylor Apr 13 '17 at 23:56
1
$\begingroup$

The simple test would be to check whether the standard deviation of the data set is higher than the mean, i.e. overdispersion. This is a common phenomenon in count data. It would indicate that the distribution is not Poisson, in which case often negative binomial distribution is suggested instead.

Another way is to use the standard $\chi^2$ goodness of fit test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.