1
$\begingroup$

Assume we have some Poisson process that produces events. In a given year we have counted $N$ of these events.

Further assume that for some reason we need to report a monthly rate instead of this yearly number and also the (estimated) standard deviation in this monthly rate.

Clearly, the monthly rate is $N/12$. Now, the question is: What is the standard deviation in this monthly number? We have two contradicting views on this.

Alice maintains that, since the monthly number ($X$) is just a scaled version of the yearly figure ($Y$), one could just apply the scaling rule for variances.

Then, with $X = Y/12$ it follows that $\rm{Var}(X) = \frac{1}{12^2}\rm{Var}(Y)$ and hence the standard deviation of the monthly figure is 1/12 of the standard deviation of the yearly figure. The latter standard deviation is $\sqrt{N}$ as this is a Poisson process. So, we have $\sigma_{X}=\sqrt{N}/12$.

Bob, on the other hand, argues that the results for each month are generated by a Poisson process with a parameter that is scaled by 12. This follows from the rule w.r.t. the sums of Poisson distributed variables. So, with $Y\sim \rm{Pois(N)}$ it follows that $X\sim \rm{Pois(N/12)}$. Clearly, $\sigma_{X}$ is just the standard deviation of such a Poisson process, which is the square root of its rate parameter. Therefore, $\sigma_{X}=\sqrt{N/12}$.

Although the means resulting from Alice's and Bob's reasoning are the same, we've got a factor of $\sqrt{12}$ between their respective standard deviations. Who is right here, Alice or Bob?

Note: The standard deviation of this monthly number is to be understood as the (theoretical) standard deviation of future determinations of this monthly number generated by the same, assumed Poisson process.

$\endgroup$
  • $\begingroup$ Neither is "correct" without more information. The first approach is giving the average monthly events. If there are $N=120$ events in a year, this assumes that there are $10$ events a month on average. This ignores the fact that the number of events per month can differ. The second approach takes this into account. It depends on your goals. $\endgroup$ – knrumsey - Reinstate Monica Sep 30 at 20:54
  • $\begingroup$ Reporting a monthly rate: I suppose it makes sense to report $N/12$. It is unclear to me what the standard deviation means for this quantity, since $N$ is observed and no longer random. Why are you interested in reporting a standard deviation, and what exactly do you want this to mean? $\endgroup$ – knrumsey - Reinstate Monica Sep 30 at 21:07
  • $\begingroup$ The reported standard deviation would be the (Theoretical) StDev of repeated measurements of this monthly number. $\endgroup$ – Sjoerd C. de Vries Sep 30 at 21:21
4
$\begingroup$

It's important to be very specific about what is going on. Your description is somewhat ambiguous, which is why you're confused.

If you observe a homogeneous Poisson process whose annual event rate is $\lambda$ for one month (pretending all months are the same length, which they really aren't), the number of events per month would be Poisson with rate $\lambda/12$. The population variance of that month's number of events would be $\lambda/12$.

Let $N$ be the number of events in a homogeneous Poisson process in a year. If you observe $n$ events, then $\hat{\lambda}=n$ and $\widehat{\text{Var}}(\hat{\lambda})=n$.

Let the number of events in a specific future one twelfth of a year be $M$. Then let $\widehat{M}$ be your predicted number of events in that "month" using the obvious estimator ($\widehat{M}=n/12$).

(i) If you're estimating the Poisson parameter for a future month, your estimate of the variance of that parameter estimate is (n/144). $\text{Var}(\widehat{M}|N=n)=n/144$.

(ii) If you're predicting the number of events in a future month, your estimate of the variance in the observed number of events in the future month will be the predicted Poisson rate (n/12). That is $\widehat{\text{Var}}(M|N=n)=n/12$.

However, if you're interested in how far out your prediction of that future count might be, you will want to estimate the variance of the prediction error $\widehat{\text{Var}}(M-\hat{M}|N=n)$, which is the sum of the two ($n/12+n/144$).

(Indeed, compare with the law of total variance, where the term in (i) is the variance of a conditional expectation and the term in (ii) is in effect the expectation of a conditional variance.)

$\endgroup$
  • $\begingroup$ Thanks, but could you perhaps further explain the step going from the expected value of n/12 (which everyone agrees on) to the variance of n/144 in your last line? This seems to be precisely the crux in the discussion between Alice and Bob and I'm not sure I can follow you there. $\endgroup$ – Sjoerd C. de Vries Oct 1 at 8:19
  • $\begingroup$ I've attempted to more clearly distinguish the possibilities. As I said, you were ambiguous about the circumstances. You have to be clear about what's going on in the month. Is this a future month whose observed count you're predicting based on the number of events over a previous year? Or are you trying to estimate the Poisson parameter for the month? Or is it something else? Where do these two arguments come from? The context might be clearer there. $\endgroup$ – Glen_b Oct 1 at 8:27
  • $\begingroup$ OK, what I have is a report from Alice that has this single number in it that is the one-year count of events. That is basically all the data that goes into this. For some reason (let's say to make it easier digestible by its intended public), Alice wants to transform this into a monthly number, and provide some notion of the possible measurement error in it, all based on the assumed Poisson distribution of the process that generated this single one-year count. I'm afraid this is as far as I can go in explaining the context. Does that help? $\endgroup$ – Sjoerd C. de Vries Oct 1 at 8:47
  • $\begingroup$ "monthly number" is ambiguous. What does the monthly number represent? My present answer discusses two possibilities: (i) an estimate of the monthly rate based on a year of data or (ii) an estimate of the actual number that will occur in a month (and then goes on to add a third, just in case - but I don't think that's what you want). It sounds like Alice was trying to do (i) but Bob is talking about (ii). You can't say "who is right" until you specify which of the two things was actually required. $\endgroup$ – Glen_b Oct 1 at 8:56
  • $\begingroup$ I think what Alice wants to convey is the number of events that one could reasonably expect to see in 1/12 of a year, based on this single year count. The reported sd would then be used to estimate the deviations from that number that one could reasonably expect. But I believe I start to see your point how one's view on what this number represents makes a difference, and since the intended public will never go as deep as this discussion, I feel the whole use of this 'monthly number' will be confusing, if not misleading, and probably should be avoided. Thanks very much for pointing this out. $\endgroup$ – Sjoerd C. de Vries Oct 1 at 9:40
2
$\begingroup$

You have for the number of counts:

  • Counts per year: $$Y \sim Pois(\lambda)$$
  • Counts per month: $$X \sim Pois(\lambda/12)$$

But...

  • Counts per month (average for 12 months) $$Y/12 \nsim Pois(\lambda/12)$$ or $$\frac{X_1+X_2+...X_{12}}{12} \nsim Pois(\lambda/12) $$

If you divide the counts over a year by twelve then you do not get a variable that corresponds to the counts for a particular individual month, but instead you get an average over twelve months.


The Poisson distribution is only to be used for the raw number of counts. It is not true for any derived (scaled) number. So a term like 'counts per T' should be used very carefully. The Poisson distribution describes 'counts' and not 'counts per T'.


Bob was right in stating that $\text{Var}(X) = \frac{\lambda}{12}$. However when you take the mean of twelve of those variables (which is what Alice computed) then you will get:

$$\text{Var} \left( \frac{1}{12} (X_1+X_2+...X_{12}) \right) = \frac{1}{12}\frac{\lambda}{12} = \frac{\lambda}{12^2}$$

and the standard deviation, $\sigma = \frac{\sqrt{\lambda}}{12}$, corresponds to Alices number.


for some reason we need to report a monthly rate instead of this yearly number and also the (estimated) standard deviation in this monthly rate

You can report

  • an estimate for the mean monthly rate and the estimated standard error for that estimate.

But note that this will be different from

  • the standard deviation of that monthly rate.

The variance of a distribution and the variance of an estimate for the mean of that distribution are not the same.

(This occurs very often that some people report figures with very tiny error bars. That makes it appear as if the difference between two cases is very small. But, what those people only did is show that they can estimate the means very precise and show those are different, but this does not mean that the differences between the groups is so large. Often it is also confusing/ambiguous what the reported variation/error bars mean.)

$\endgroup$
1
$\begingroup$

what I understand is that you know just the number $N_{\rm y}$ of events that occurred in a year time interval $t_{\rm y}$. From that, you would like to report an estimation of how many events $N_{\rm m}$ occurred in each month time interval $t_{\rm m}$, and importantly, what is the variability that one would expect for that estimated $N_{\rm m}$.

Answer: The approach would be to estimate the rate $\lambda$ by its maximum likelihood estimator, $\hat{\lambda} = N_{\rm y}/t_{\rm y}$. If we were talking about a prediction for next month, then assuming that the rate parameter $\lambda$ does not depend on time you can use $\hat{\lambda}$ and obtain

$$ \left< N_{\rm m} \right> = N_{\rm y}/12 $$ $$ \textrm{std}(N_{\rm m}) = \sqrt{ N_{\rm y}/12 } $$

However, it doesn't matter if we are dealing with a prediction for next month or a prediction of what already happened but you don't know, so this result is the one you are looking for.

Note: if you have a prior knowledge for reasonable values of $\lambda$, you could go further and use Bayesian statistics, which actually modify the form of $\textrm{std}(N_{\rm m})$.

Why the transformation of variables is not correct? Because in that case you are not dealing with the statistics of the monthly results, you are still dealing with the statistics of a yearly result but you consider that each individual event which counts, instead of as one unit $N=1$, as $N=1/12$. In that case you get estimations of the same order for the average, that's clear, but the variability is not the right one for a month.

Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.