# How do I figure out how many items I should expect to exceed my error threshold?

Let me explain what numbers I'm working with, then I'll explain the problem.

1. I calculate the 99th percentile forecast error threshold across all items for the past year at a store.
2. Then I report the number of items that, in the past two weeks, had an error greater than that threshold. (It doesn't matter if it was one day or fourteen, they get flagged if they had a forecast error greater than the threshold.)

Ok, so that's that. The problem is, I want to know, statistically, how to calculate the expected number of items that will flag given the generic details of my calculation method, so I can see when we have deviations.

I thought about Poisson since this is basically an arrival rate problem, but that feels circular to me in my head since λ is sort of what I'm after to begin with, and I'm not sure what I'd use for the rest of the parameters anyway.

My manager thinks it will be 1%, but I think that's naively projecting the 99th percentile deal, which was calculated across a year, into the arrival rate for a 2 week period going forward. If that were true you'd expect 1% of items to trigger for one day, one week, one month, which obviously cannot hold, right? More items would trigger with more time, and fewer with less. I'm not sure if it is relevant if the two week period is part of the data used to calculate the thresholds or not?

What's the statistical way I can determine the expected proportion?

clarification edit: I've run this on my data already, so I know what my specific case's baseline is. But I can't figure out how you would calculate the expected number of items to trigger in the general case. I just want to know how you would go about doing that.

• Part of the complexity is that you have time series, and errors over time within a product may well be highly autocorrelated, because some products are just harder to forecast than others. In the extreme case where forecast errors are constant over time for all products, the same 1% of products will trigger every single time. If autocorrelation is lower, you may get different products triggering every time. Worst case, each day of your 14 days completely different products trigger, so now you have 14%. You can't get more precise in general without digging into your actual data. Commented Apr 17 at 13:40
• I'm sort of asking for the general case though, just purely mathematical, ignoring the complexities of my data. If you have that fact set, n items, 99th percentile year thresholds, 2 week arrival period, etc Commented Apr 17 at 13:52

## Stating and Simplifying the Problem

To simplify things, let's assume that the total number of items, $$n$$, never changes. Let $$X_{ij} = 1$$ if item $$i$$ exceeds the threshold on day $$j$$. Let $$Y_i = \max_{j=1,\ldots, 14}\{X_{ij}\}$$ be an indicator for whether item $$i$$ exceeds the threshold at any point during the two week period. You are interested in the proportion $$p = E\left[\frac{1}{n}\sum_{i=1}^n Y_i\right] = \frac{1}{n}\sum_{i=1}^nE(Y_i),$$ where the final equality holds regardless of dependence (by linearity of expectation). It is also reasonable to assume (in many cases, at least) that $$E(Y_i)$$ is the same for all $$i=1,\ldots, n$$, and so we have $$p = E(Y_1)$$.

## Finding $$E(Y_i)$$

Since $$Y_i$$ is a binary random variable, we only need to find the probability that it equals $$1$$. That is... $$E(Y_i) = 1\times P(Y_1 = 1) + 0\times P(Y_i = 0) = P(Y_i=1).$$ We can find this probability using a common theme in statistics; essentially that $$Y_i=0$$ if and only if $$\{Y_{ij}=0\}$$ for all $$j$$. If you think about this in terms of your problem this should be obvious!$$^\ast$$ \begin{align*} p &= 1 - P(Y_i = 0) \\ &= 1 - P\left((Y_{i1}=0) \cap (Y_{i2} = 0) \cap \ldots \cap (Y_{i,14}=0)\right). \end{align*}
In general, this probability can be difficult to find, as it depends on the autocorrelation of the random variables and the statistical model you rely on.

### Indepdence

If you are willing to assume independence across time (probably unrealistic), then we get $$p = 1 - P(Y_{i1} = 0)^{14} = 1 - 0.99^{14} = 0.1313.$$

### Dependence

In the general case, it is hard to say much about the proportion without problem/model-specific information (this is @StephanKolassa's main point). We can say something about the worst case, using Boole's inequality: \begin{align*} p &= 1 - P\left((Y_{i1}=0) \cap (Y_{i2} = 0) \cap \ldots \cap (Y_{i,14}=0)\right) \\ &= 1 - \left(1 - P\left((Y_{i1}=1) \cup (Y_{i2} = 1) \cup \ldots \cup (Y_{i,14}=1)\right)\right) && \text{DeMorgan's Law} \\ &= P\left((Y_{i1}=1) \cup (Y_{i2} = 1) \cup \ldots \cup (Y_{i,14}=1)\right) \\ &\leq \sum_{j=1}^{14}P(Y_{ij} = 1) &&\text{Boole's inequality} \\ &= \sum_{j=1}^{14}0.01 = 0.14. \end{align*}
But knowing the "worst case" isn't particularly useful. We can construct a dependence structure to give any value of $$p$$ between $$0.01$$ and $$0.14$$.

$$^\ast$$ An item fails to exceed the threshold in a two week period if and only if it fails to exceed the threshold every day during the two week period. Duh.

• Thank you very much for the expanded explanation and reference of DeMorgan's and Boole's! (I don't have enough karma for my upvotes to show.) All helpful information. Commented Apr 18 at 2:42
• I figured out how to code up Boole's, so I can calculate my item likelihoods across time and plug them in and get a specific-case answer, appreciate it! Commented Apr 18 at 2:57

Part of the complexity is that you have time series, and errors over time within a product may well be highly autocorrelated, because some products are just harder to forecast than others. Or not. It absolutely depends on your situation.

In the extreme case where forecast errors are constant over time for all products, the same 1% of products will trigger every single day.

If autocorrelation is lower, you may get different products triggering every time. Worst case, each day of your 14 days completely different products trigger, so now 14% of your products trigger at some day during your two weeks evaluation period.

You can't get more precise than "somewhere between 1% and 14%" in general without digging into your actual data. (Yes, complete independence means that your expected percentage can be calculated a bit more precisely, but I don't think that adds a lot of enlightenment over "look at your actual data".)

• I see, so 14% is a sensible upper bound, because you'd expect 1% of the items (different each day) to hit the 99th percentile threshold every day in the worst case. I should clarify: I'm after the expected percentage to learn more about stats and how you'd solve this problem from that perspective. I don't really plan on using this to help my job work, but to learn. As you said, "look at your data" - I ran it for a whole year as a test, so I know what to expect from my data. But I want to know the stats answer for what the expected baseline and/or lower bound would be. Commented Apr 17 at 14:19
• Per my answer, the expectation will depend mostly on the autocorrelation of errors within each product, and the general level of errors within each product. If you can make assumptions here, you can estimate the expectations through simulation. (There may or may not be a numerical way of solving the issue.) Commented Apr 17 at 14:28
• I don't see how you are getting $14\%$. The calculation is a bit more complicated than that. I show in my answer that the fully independent case gives $1 - 0.99^{14} \approx 0.1313$. So your $14\%$ rule is a good approximation, but not exact. Can you expand on your answer? I do agree with you on everything else! (: Commented Apr 17 at 18:57
• @knrumsey: that is exactly the point of my last sentence in parentheses. Yes, you are right that 14% will not hold under independence. (But given a sufficiently unfortunate dependence structure between the errors within and between products, we can "achieve" 14%.) My point is that IMO, the "general" discussion is rather fruitless. Commented Apr 17 at 21:04
• "given a sufficiently unfortunate dependence structure... we can achieve 14%". This is the part I was wondering about, but I figured out how to show it. I agree that there's not much more to say, however, without knowing more about the data/model. +1! Commented Apr 17 at 22:34