Detecting outliers in a multiple time-series

Question

I have some broker prices incoming in real-time for several products. Sometimes a broker makes a typo and sends a wrong price, or my parsing engine assigns the price to the wrong product - these are the two ways an outlier can occur.

I wish to remove the outliers, in real-time as they come in. I cannot just look at each time-series of each product individually since they are non-stationary and exhibit jumps as the market changes. My idea was to compare the incoming price not only with the past prices of the product it got assigned to, but also other products in order to see whether:

1. Whether the price 'fits' well into the assigned product
2. Whether the price fits better into another product with a similar name (probably a parsing error)
3. If the new price was assigned to a product but it is quite different in value than recent prices - check with other related products if they all also jumped, then the new price is probably fine as the whole market jumped.

I have these ideas but cannot put them together in terms of what outlier techniques to use. I prefer simpler methods rather than cutting-edge unsupervised ML algos since it is important there is explainability.

Advice is appreciated.

Answer 1

I am not sure what the notion of "jumps" refers to and how irregular they can be. For what follows, I simply presume that each of your $m$ price time series $x_t^{(i)}, i=1,\ldots, m$ is sufficiently well behaved such that time series predictions are possible.

So my suggestion is to learn a time series model for each of your prices (e.g. ARIMA) and to obtain a prediction distribution $p_t^{(i)}(x)$ for each of them. Thus, you get a time-varying family of $m$ prediction distributions.

Next, for each price $x^{(i)}(t)$, you check the ratio $\chi_t(i)$ between the largest value $p_t^{(k)}(x_t^{(i)})$ for $k=1,\ldots,m$ and the value $p_t^{(i)}(x_t^{(i)})$: $$ \begin{align} k^\ast_t(i) &:= argmax_k\{p_t^{(k)}(x_t^{(i)})\}\\ \chi_t(i) &:= \frac{p_t^{(k^\ast_t(i))}(x_t^{(i)})}{p_t^{(i)}(x_t^{(i)})}. \end{align} $$ The ratio $\chi_t(i)$ is then your outlier score. If $k_t^\ast(i) = i$, then $\chi_t(i) = 1$, otherwise it is larger than $1$. You now have to choose a threshold for this score above which an outlier alarm should be triggered.