3
$\begingroup$

I'm trying to figure out how to test if variables that I think follow a Poisson distribution are independent (which is a requirement of the Poisson distribution) -- or it it matters.

Let's assume I count the number of shoppers, X, that come into my store in every 6-hour timeframe. After a month of doing that, I want to determine which of those timeframes saw a significantly smaller or larger number of shoppers. I can calculate a lambda (average rate of X/timeframe), plot a time series, and compare the result to a poisson distribution with the same lambda -- it's likely going to be similar.

But, for Xi to be Poisson-distributed, they need to be independent. In most examples of Poisson-related work on time series (e.g., http://pymc-devs.github.io/pymc/tutorial.html), the variables are more obviously independent: mining disasters, car crashes, text messages received, etc. In an example like mine, they might not be; for example, a large number of shoppers going into a store at once could motivate others to go into the store as well.

How do I test for independence, beyond making assumptions about the situation? A chi-square test seems most obvious, but there's no degrees of freedom to run a chi-square test.

[It probably doesn't matter for this question, but my goal is to run a very similar analysis to the one detailed in the pymc link -- to set up a Markov Chain model that shows when a likely change in frequency/lambda occurred. That model has the same independence requirement though.]

$\endgroup$

2 Answers 2

3
$\begingroup$

I'm trying to figure out how to test if variables that I think follow a Poisson distribution are independent (which is a requirement of the Poisson distribution)

You seem to be conflating the Poisson process with the Poisson distribution here.

For the Poisson process to really be a Poisson process, it has to be independent, and then if the rate is constant (and the remaining assumptions are true), you'll get Poisson observations (the count of events in the process over a given interval will be Poisson). However that doesn't imply that observations that are Poisson-distributed are automatically independent; it's possible to construct situations where Poisson data over time is serially dependent, say (and it's possible to find data where such a dependent-Poisson model is a plausible description of the data).

But, for Xi to be Poisson-distributed, they need to be independent.

Indeed, we can't even read this as confusion over Poisson process vs Poisson distribution -- this is just not true.

In most examples of Poisson-related work on time series (e.g., http://pymc-devs.github.io/pymc/tutorial.html),

Most you're aware of, perhaps. I don't have data from which to conclude what might be most common.

the variables are obviously independent: mining disasters, car crashes, text messages received, etc.

Actually, I can see several ways that there might be dependence in those (partly depending on what you condition on). Independence might be a plausible model, but that doesn't make it necessarily true.

In an example like mine, they might not be; for example, a large number of shoppers going into a store at once could motivate others to go into the store as well.

And, for example, one mine disaster might prompt safety reviews in others, making them negatively dependent (a disaster might well make further disasters less likely for a period), and similarly there are possible dependencies for the others. I don't see how your assertion that they're obviously independent is justified.

How do I test for independence, beyond making assumptions about the situation?

But you cited an obvious assumption to consider - that there could be serial dependence. That's reasonably easy to test for, in a number of ways.

If you want to test for every possible kind of dependence you have some problems, but there's no great difficulty checking for simple forms of serial dependence.

The first thing to consider would be a plot of $y_t$ vs $y_{t-1}$. Here are data that are (by construction) identically distributed Poisson (all with $\lambda=240$):

$\ $serially correlated Poisson data

The sample Pearson correlation is 0.543 (the Spearman correlation is 0.526, Kendall's tau is 0.371). You might - as an example - formally test a null of independence against the alternative of lag-1 dependence via a permutation test, but if you're just trying to check model assumptions or find a good model, formal testing is probably not the most effective choice.

$\endgroup$
4
  • $\begingroup$ You're right, I was conflating here---I've been reading about the Poisson process, and applied its assumptions to the Poisson distribution. I made some changes in the question to clarify for others who come across it. Serial dependence is exactly what I was looking for. If I read you correctly, I'd have to run a correlation test n-1 times: between the first two timeframes, the second and third, the third and fourth, etc. That shouldn't be too difficult to do with a simple iterating program, but are there more efficient ways of testing serial dependence across all timeframes? $\endgroup$ Commented Oct 6, 2014 at 1:38
  • $\begingroup$ If I understood your situation right you have one number per time-frame and a series of observations across time-frames. As such you'd (at least to begin with) have only one correlation across the whole set (though you might later specify other subsets of time). You might look, however, at different lags (with, presumably more than 1 six hour period per day, we might see higher day to day correlation at lags higher than 1), suggesting looking at an ACF or a PACF (though the usual 95% CI lines that are often supplied on them won't be quite right), for example, perhaps out to lag 4 or so. $\endgroup$
    – Glen_b
    Commented Oct 6, 2014 at 1:57
  • $\begingroup$ I don't follow how the acf solution will work; it's only meaningful for a stationary process, which this will not be if the underlying Poisson process has a varying rate. $\endgroup$ Commented Oct 6, 2014 at 3:06
  • $\begingroup$ @Nir It's a diagnostic tool rather than a cut-and-dried solution to anything; certainly if the rate is varying that would cause autocorrelation on its own, just as it would in any other time series. My initial concern in my answer was with explaining that you can explore the dependence structure and one particular kind of dependence structure is obvious to look for with this situation, and to talk about ways of looking at that structure. I didn't anywhere seek to imply that the varying rate wasn't an issue, but my simple message wasn't understood as it was, without adding complexity. $\endgroup$
    – Glen_b
    Commented Oct 6, 2014 at 3:11
0
$\begingroup$

I've come across this before. Making zero assumptions, I don't think it's possible to do anything (at least anything good). The reason is that even if it is Poisson distributed, it can (and likely is) rate varying poisson. If you are really making zero assumptions, then the rate is is a function of t, and then you basically only have one sample on N potentially different distributions, each a Poisson distribution with a different parameter.

There's basically two reasonable assumptions you can make. One is to limit the time dependence somehow. Most obviously, assume that the rate depends only on the time of day, and not on the day. Then, if your hypothesis is correct, you have 4 independent poisson variables (with potentially different rates), and as many samples on each variable as you have days. Then you can do lots of stuff. The other is if you have access to data sampled at a much higher rate than every 6 hours, you can try to assume that the rate changes slowly compared to how fast you collect data, and do something with that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.