I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question.

I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based on Product Sales Forecast which I run through a model which has various tasks (such as changing items on a shelve or selling a car alarm for example) binned into timed categories, e.g

Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations.

Now, as, for each iteration of this model I've run, there are always outliers (read mistakes), such as Store Y has no productVolumes to ProductGroup X in Week 22, which is a mistake from the finance team.

My senior has said this has always been the case for the 8 or so years he's been with the company.

Now, in my head, I assume I would be able to find outliers based on historical data using some sort of statistical method, however, I have no idea where to start, my data is as follows and is around 10 million rows of data.

    import pandas as pd
    import numpy as np

    data =  [19,21,24,18,3]
    pg = ['PG','ZF','AA','GG','ZF']
    location = ['AA_1','AA_1','AA_2','AA_2','AA_2']
    weeks = [1,1,2,2,2]

    df = pd.DataFrame({'Location' : location,
                'productGroup' : pg,
                'Week' : weeks,
                'productVolumes' : data })
      Location  productGroup    Week    productVolumes
    0   AA_1    PG  1   19
    1   AA_1    ZF  1   21
    2   AA_2    AA  2   24
    3   AA_2    GG  2   18
    4   AA_2    ZF  2   3

What would be a suitable method to apply to this problem to help me identify outliers?

The outliers would be a massive variation from the last forecast (which would have been corrected by the time we launched it) This would also include data that is missing.

  • $\begingroup$ This to me is not clear, can you explain it: "Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations." How does an object take 30 seconds? Does the forecasting take 30 seconds? What do you mean by allowing time? $\endgroup$ – Zhubarb Jun 3 '19 at 15:02
  • $\begingroup$ @Zhubarb it means that the time allocated for the sale of the product is 30 seconds, which is what I am forecasting (time). Just for clarity, finance tell me they are forecasting 20 shoes to sell, I know, from doing timed studied that it takes my retail team 120 seconds to sell this item, the time allows for advice, trying on a shoe and selling the item, so I will run the sales forecast through my labour model which will output 120 *20 which is 2,400 seconds or 40 minutes. The finance forecasts are broken down into week, location level by productGroup $\endgroup$ – Manakin Jun 3 '19 at 15:13
  • $\begingroup$ Thanks, is Product volume the forecasted numbers to sell in that particular week, e.g. week 1? Is week 1 the first week of January in that calendar year? $\endgroup$ – Zhubarb Jun 3 '19 at 15:21
  • $\begingroup$ @Zhubarb yes that's correct, they are broken down into 52 weeks starting in April $\endgroup$ – Manakin Jun 3 '19 at 15:49

There are various outlier / novelty detection algorithms available that you can use for this. You could also "hack" some clustering algorithms to attain the same goal. You will need to check that the algorithm supports the input of categorical variables like location, week of the year, product group, etc. This requirement eliminates some obvious options like Elliptic Envelope (1) where the assumption is that your data follows a Gaussian distribution.

The categorical features result in a very high dimensional feature space, e.g. week of the year [52] * product group [10,000] * Location [500], etc. Given you have 10M+ observations to work with, you may not need to do anything, but some aggregations to less granular categories, for instance month of the year, or coarser post codes, etc., may be necessary. When week or month of the year is explicitly included in the model, seasonality should not be an issue, but I am happy to stand corrected. My experience is that time series deseasonalisation in these types of applciations requires great care and I have not had too much success with it.

Among the obvious algorithm candidates you can use are One-class SVM (2), Isolation Forest (3), and Local Outlier Calculation (4). I have attached the information on papers on all, and a quick read-through would prove useful. In addition to novelty detection algorithms, you can also utilise some clustering algorithms to detect for outliers. DBScan (5) is one of my favourites. By tweaking the min sample (for a cluster) and epsilon parameters, you can make the algorithm assume a single cluster and highlight any outliers as "noise observations".

Bear in mind that DBScan takes a long time to run and in terms of computation time it may struggle with 10M+ observations (still give it a go). The other algorithms should not have a problem as long as the data fits in your memory (no big data solution here).

Another general note is that, for all of the above, you will need to enter an estimated "outlier fraction" as a model input. So none of these is actually fully "unsupervised".

With regards to including missing data as outliers, not sure if this makes sense or whether it is practically possible. Your remark on this is not clear to me.

Finally, this link should give you a good start in terms of how to use some of these novelty detection algorithms, already implemented in Python. Also, the implementation for DB-Scan is here.

  • (1) Rousseeuw, P.J., Van Driessen, K. “A fast algorithm for the minimum covariance determinant estimator” Technometrics 41(3), 212 (1999)
  • (2) http://www.jmlr.org/papers/volume2/manevitz01a/manevitz01a.pdf
  • (3) Liu, Fei Tony, Ting, Kai Ming and Zhou, Zhi-Hua. “Isolation forest.” Data Mining, 2008. ICDM‘08. Eighth IEEE International Conference on.
  • (4) Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000, May). LOF: identifying density-based local outliers. In ACM sigmod record.
  • (5) Ester, M., H. P. Kriegel, J. Sander, and X. Xu, “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise”. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland, OR, AAAI Press, pp. 226-231. 1996
| cite | improve this answer | |
  • $\begingroup$ Thanks for this, this might be out of scope regarding this question, would it be possible to have some sample python code for one of the algorithms? The missing data would be that Location X has no product volumes for group Y. I'll read through all these, thank you my man. $\endgroup$ – Manakin Jun 3 '19 at 17:06
  • $\begingroup$ Ignore me, just went through the sci kit learn and this is exactly what I needed! $\endgroup$ – Manakin Jun 3 '19 at 17:38
  • $\begingroup$ my sincerest apologies, this was my first time issuing a bounty and I wrongly assumed the bounty would be given the moment I chose an accepted answer! Thanks again! I'm still working on implementing and understand all the above, but you've given me direction which is invaluable. Thanks again. $\endgroup$ – Manakin Jun 9 '19 at 10:21
  • $\begingroup$ I am glad it helped :) $\endgroup$ – Zhubarb Jun 9 '19 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.