1
$\begingroup$

I have some broker prices incoming in real-time for several products. Sometimes a broker makes a typo and sends a wrong price, or my parsing engine assigns the price to the wrong product - these are the two ways an outlier can occur.

I wish to remove the outliers, in real-time as they come in. I cannot just look at each time-series of each product individually since they are non-stationary and exhibit jumps as the market changes. My idea was to compare the incoming price not only with the past prices of the product it got assigned to, but also other products in order to see whether:

  1. Whether the price 'fits' well into the assigned product
  2. Whether the price fits better into another product with a similar name (probably a parsing error)
  3. If the new price was assigned to a product but it is quite different in value than recent prices - check with other related products if they all also jumped, then the new price is probably fine as the whole market jumped.

I have these ideas but cannot put them together in terms of what outlier techniques to use. I prefer simpler methods rather than cutting-edge unsupervised ML algos since it is important there is explainability.

Advice is appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

I am not sure what the notion of "jumps" refers to and how irregular they can be. For what follows, I simply presume that each of your $m$ price time series $x_t^{(i)}, i=1,\ldots, m$ is sufficiently well behaved such that time series predictions are possible.

So my suggestion is to learn a time series model for each of your prices (e.g. ARIMA) and to obtain a prediction distribution $p_t^{(i)}(x)$ for each of them. Thus, you get a time-varying family of $m$ prediction distributions.

Next, for each price $x^{(i)}(t)$, you check the ratio $\chi_t(i)$ between the largest value $p_t^{(k)}(x_t^{(i)})$ for $k=1,\ldots,m$ and the value $p_t^{(i)}(x_t^{(i)})$: $$ \begin{align} k^\ast_t(i) &:= argmax_k\{p_t^{(k)}(x_t^{(i)})\}\\ \chi_t(i) &:= \frac{p_t^{(k^\ast_t(i))}(x_t^{(i)})}{p_t^{(i)}(x_t^{(i)})}. \end{align} $$ The ratio $\chi_t(i)$ is then your outlier score. If $k_t^\ast(i) = i$, then $\chi_t(i) = 1$, otherwise it is larger than $1$. You now have to choose a threshold for this score above which an outlier alarm should be triggered.

$\endgroup$
2
  • $\begingroup$ Interesting. Could I ask what do you refer to when you say 'prediction distribution'? $\endgroup$
    – MilTom
    May 3 at 12:55
  • $\begingroup$ @MilTom Time series models allow you to predict, forecast, the next values of the time series. This prediction is a random variable that has a distribution. Time series libraries have functionality that will provide you with this distribution. $\endgroup$
    – frank
    May 4 at 3:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.