5
$\begingroup$

This question stems from the comment section of this question: Bus wait time under Poisson distribution, where it seems that

The properties of the Poisson don't make sense for times because the units don't match up. Standard deviation of times should be in minutes but the Poisson has variance = mean. Change the units to seconds or hours and you lose variance=mean, so it can't be Poisson.

However, my response was that

Wouldn't this imply the Poisson distribution cannot be model anything that has units? If the number of policies an insurance agent sells per week follows a Poisson distribution with mean of 3, the standard deviation would be $\sqrt{3}$ policies, and variance would be 3 policy squared, which seems legitimate?

Furthermore, in the original question, I inquired about what would happen if we discretize time, rounding to the nearest minute, so that it is at least a discrete distribution taking on non-negative values - falling into the domain of Poisson distribution.

My question is: what is preventing modeling of wait time in that problem with the Poisson model?

$\endgroup$

1 Answer 1

2
$\begingroup$

Let us handwave away the discrete/continuous issue and pretend that the times have been discretized by a hypothetical researcher, and they're in units of minutes.

So let us say that when expressed in whole minutes (whether rounded off or rounded down, but let us imagine rounded down for he moment), so there are empirical proportions for 0 minutes (representing 0 seconds to 59.999... seconds), 1 minute (representing 60seconds to 119.999... seconds), and so on.

Imagine also that there's plenty of data.

Further let us imagine that (somehow) it turned out that to this researcher the set of empirical probabilities do look close to a Poisson distribution with some unspecified parameter.

Now we imagine that another person takes the same data and stores it in seconds rather than minutes. They too round down their times into bins, recording "0" for times between 0 seconds and 59.999... seconds), "60" for times between 60 seconds and 119.999... seconds and so on.

These two people are in complete agreement about what they observe -- all they need to do is communicate the fact of the units they're working in and everything converts nicely between them. If one of them calculates a mean or a standard deviation, and the other converts it to their own units, it matches their calculation. In fact, everything they do converts over perfectly.

But now consider the first person's thought that that the empirical distribution is a good match for a Poisson distribution. Does this make sense to the second person?

Well, no, not at all. The variance is about 60 times too large for it to be a Poisson. A third person, who worked in hours (rounded down to completed $\frac{1}{60}$ths of an hour) would say that the variance was much too small. A fourth that worked in nanocenturies would say it was somewhat out, but not by a factor as big as 60.

This appearance that the data were "Poisson" was illusory then -- a happenstance outcome that is purely a matter of the choice of the units of scale.

A quick simulation to illustrate:

> x=rpois(1000,5) # researcher's data, in minutes
> var(x)/mean(x)
[1] 0.9916196     # looks reasonable for a preliminary check
> y=x*60          # 2nd person's data, in seconds
> var(y)/mean(y)
[1] 59.49717    # Clearly not Poisson!

This units problem is not present when the data are counts. Yes, we're counting something but the counts themselves do not have units, and all participants will be able to agree about the evidence on the suitableness (or otherwise) of the Poisson for their counts (they might interpret the information differently - one might say it's close enough and the next might think it's not really close enough - but they agree about the information that is in the data in relation to that choice). They don't have a unit-conversion problem when the data are counts.

So now let us imagine the original researcher is cognizant of the measurement-unit issue. If they don't claim that the Poisson distribution is a good model, what should they claim instead, that their companions could all agree on the evidence for, after converting units?

The appropriate model for measurements that (somehow) look Poisson-like that would actually survive unit-conversion is a scaled Poisson (quasi-Poisson) model. That is, if $N\sim \text{Pois}(\lambda)$, then let $Y=\phi N$ for $\phi$ some constant. The units are taken into the dispersion parameter, $\phi$. So we have$\text{E}(Y)=\phi\lambda$ and $\text{Var}(Y) = \phi^2\lambda$. Now known facts about how variances scale under change of units actually work.

When one person converts another person's information into their own units, they simply get a different $\phi$, but their $\phi$ parameter is expressed in their specific choice of units ($\lambda$ is unitless and the same for everyone). Consequently everyone understands what the various people's 'numerically' different $\phi$ is telling them -- they are all expressions of the same scale factor, simply in different units.

Our researcher's claim is then best expressed something like this: "My data look like they might reasonably be modelled by a scaled Poisson (a.k.a. quasi-Poisson) with $\lambda$ around $5$ and $\phi$ near $1$ minute (say). The second person would agree, saying "sure, that looks right -- it does look more or less like a quasi-Poisson where $\lambda$ is indeed around $5$ and $\phi$ is about $60$ seconds".

If this were real data, the fact that it comes out quite so near 1 is somewhat implausible (it's a heck of a coincidence) but not actually impossible. We could handwave away that roundness for an example question, which is where this question originated.

There's continuous models that might make a better starting point for the elapsed time, but at least framed the way we have it now, the model is no longer patent nonsense.


A warning. Beware of doing this quasi-Poisson thing with monetary amounts, where inflation can come in; if you use GLMs on such data where there's observations across time you will get into trouble.

When the values are Poisson-ish counts times some constant, it all works fine. When they're counts times a quantity that is not constant, it's simply no longer a GLM.

I can't resist a somewhat snarky aside on this issue: I see a common error amongst certain groups of professionals (I'll avoid naming the profession directly) ... using quasi-Poisson GLMs when the data are subject to inflation; this leads to changing dispersion parameters across time, taking it outside the GLM framework, so the results from a GLM won't predict right - the changing disperson is instead pulled into the model for the Poisson parameter - and the variance estimates and parameter standard errors and the relative weights to observations will be off.

[In the distant past I refereed several papers that made this exact error, even though it had been noted in their literature already. Eventually some authors submitted this same stuff - all of the papers used the exact same model - to a journal outside that area, which I was not a referee on, and more people have piled on since.]

This - and some other such scale issues, such as currency conversions at multiple rates across time - can lead to some serious errors if the people doing such calculations get into 'plug and chug' mode without adequate thought. This is impacting the financial adequacy of a number of companies to a (presently) moderate degree. Relatively low inflation has kept it from showing the position of those companies up very clearly. The current higher inflation we are seeing in the economy may start to cause considerably more serious effects over the next few years, especially as many parts of the economy become more unstable at the same time. It will be interesting to see what happens.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.