How do I figure out how many items I should expect to exceed my error threshold?

Question

Let me explain what numbers I'm working with, then I'll explain the problem.

1. I calculate the 99th percentile forecast error threshold across all items for the past year at a store.
2. Then I report the number of items that, in the past two weeks, had an error greater than that threshold. (It doesn't matter if it was one day or fourteen, they get flagged if they had a forecast error greater than the threshold.)

Ok, so that's that. The problem is, I want to know, statistically, how to calculate the expected number of items that will flag given the generic details of my calculation method, so I can see when we have deviations.

I thought about Poisson since this is basically an arrival rate problem, but that feels circular to me in my head since λ is sort of what I'm after to begin with, and I'm not sure what I'd use for the rest of the parameters anyway.

My manager thinks it will be 1%, but I think that's naively projecting the 99th percentile deal, which was calculated across a year, into the arrival rate for a 2 week period going forward. If that were true you'd expect 1% of items to trigger for one day, one week, one month, which obviously cannot hold, right? More items would trigger with more time, and fewer with less. I'm not sure if it is relevant if the two week period is part of the data used to calculate the thresholds or not?

What's the statistical way I can determine the expected proportion?

clarification edit: I've run this on my data already, so I know what my specific case's baseline is. But I can't figure out how you would calculate the expected number of items to trigger in the general case. I just want to know how you would go about doing that.

Answer 1

Stating and Simplifying the Problem

To simplify things, let's assume that the total number of items, $n$, never changes. Let $X_{ij} = 1$ if item $i$ exceeds the threshold on day $j$. Let $Y_i = \max_{j=1,\ldots, 14}\{X_{ij}\}$ be an indicator for whether item $i$ exceeds the threshold at any point during the two week period. You are interested in the proportion $$p = E\left[\frac{1}{n}\sum_{i=1}^n Y_i\right] = \frac{1}{n}\sum_{i=1}^nE(Y_i),$$ where the final equality holds regardless of dependence (by linearity of expectation). It is also reasonable to assume (in many cases, at least) that $E(Y_i)$ is the same for all $i=1,\ldots, n$, and so we have $p = E(Y_1)$.

---

Finding $E(Y_i)$

Since $Y_i$ is a binary random variable, we only need to find the probability that it equals $1$. That is... $$E(Y_i) = 1\times P(Y_1 = 1) + 0\times P(Y_i = 0) = P(Y_i=1).$$ We can find this probability using a common theme in statistics; essentially that $Y_i=0$ if and only if $\{Y_{ij}=0\}$ for all $j$. If you think about this in terms of your problem this should be obvious!$^\ast$ $$\begin{align*} p &= 1 - P(Y_i = 0) \\ &= 1 - P\left((Y_{i1}=0) \cap (Y_{i2} = 0) \cap \ldots \cap (Y_{i,14}=0)\right). \end{align*}$$
In general, this probability can be difficult to find, as it depends on the autocorrelation of the random variables and the statistical model you rely on.

Indepdence

If you are willing to assume independence across time (probably unrealistic), then we get $$p = 1 - P(Y_{i1} = 0)^{14} = 1 - 0.99^{14} = 0.1313.$$

Dependence

In the general case, it is hard to say much about the proportion without problem/model-specific information (this is @StephanKolassa's main point). We can say something about the worst case, using Boole's inequality: $$\begin{align*} p &= 1 - P\left((Y_{i1}=0) \cap (Y_{i2} = 0) \cap \ldots \cap (Y_{i,14}=0)\right) \\ &= 1 - \left(1 - P\left((Y_{i1}=1) \cup (Y_{i2} = 1) \cup \ldots \cup (Y_{i,14}=1)\right)\right) && \text{DeMorgan's Law} \\ &= P\left((Y_{i1}=1) \cup (Y_{i2} = 1) \cup \ldots \cup (Y_{i,14}=1)\right) \\ &\leq \sum_{j=1}^{14}P(Y_{ij} = 1) &&\text{Boole's inequality} \\ &= \sum_{j=1}^{14}0.01 = 0.14. \end{align*}$$
But knowing the "worst case" isn't particularly useful. We can construct a dependence structure to give any value of $p$ between $0.01$ and $0.14$.

---

$^\ast$ An item fails to exceed the threshold in a two week period if and only if it fails to exceed the threshold every day during the two week period. Duh.

Answer 2

Part of the complexity is that you have time series, and errors over time within a product may well be highly autocorrelated, because some products are just harder to forecast than others. Or not. It absolutely depends on your situation.

In the extreme case where forecast errors are constant over time for all products, the same 1% of products will trigger every single day.

If autocorrelation is lower, you may get different products triggering every time. Worst case, each day of your 14 days completely different products trigger, so now 14% of your products trigger at some day during your two weeks evaluation period.

You can't get more precise than "somewhere between 1% and 14%" in general without digging into your actual data. (Yes, complete independence means that your expected percentage can be calculated a bit more precisely, but I don't think that adds a lot of enlightenment over "look at your actual data".)