Finding Outliers in Resource Allocation Forecast Data I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question.
I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based on Product Sales Forecast which I run through a model which has various tasks (such as changing items on a shelve or selling a car alarm for example) binned into timed categories, e.g
Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations.
Now, as, for each iteration of this model I've run, there are always outliers (read mistakes), such as Store Y has no productVolumes to ProductGroup X in Week 22, which is a mistake from the finance team.
My senior has said this has always been the case for the 8 or so years he's been with the company.
Now, in my head, I assume I would be able to find outliers based on historical data using some sort of statistical method, however, I have no idea where to start, my data is as follows and is around 10 million rows of data.
    import pandas as pd
    import numpy as np

    data =  [19,21,24,18,3]
    pg = ['PG','ZF','AA','GG','ZF']
    location = ['AA_1','AA_1','AA_2','AA_2','AA_2']
    weeks = [1,1,2,2,2]

    df = pd.DataFrame({'Location' : location,
                'productGroup' : pg,
                'Week' : weeks,
                'productVolumes' : data })
    print(df)

      Location  productGroup    Week    productVolumes
    0   AA_1    PG  1   19
    1   AA_1    ZF  1   21
    2   AA_2    AA  2   24
    3   AA_2    GG  2   18
    4   AA_2    ZF  2   3

What would be a suitable method to apply to this problem to help me identify outliers? 
The outliers would be a massive variation from the last forecast (which would have been corrected by the time we launched it) This would also include data that is missing.
 A: There are various outlier / novelty  detection algorithms available that you can use for this. You could also "hack" some clustering algorithms to attain the same goal. You will need to check that the algorithm supports the input of categorical variables like location, week of the year, product group, etc. This requirement eliminates some obvious options like Elliptic Envelope (1) where the assumption is that your data follows a Gaussian distribution.  
The categorical features result in a very high dimensional feature space, e.g. week of the year [52] * product group [10,000] * Location [500], etc. Given you have 10M+ observations to work with, you may not need to do anything, but some aggregations to less granular categories, for instance month of the year, or coarser post codes, etc., may be necessary. When week or month of the year is explicitly included in the model, seasonality should not be an issue, but I am happy to stand corrected. My experience is that time series deseasonalisation in these types of applciations requires great care and I have not had too much success with it.
Among the obvious algorithm candidates you can use are One-class SVM (2), Isolation Forest (3), and Local Outlier Calculation (4). I have attached the information on papers on all, and a quick read-through would prove useful. In addition to novelty detection algorithms, you can also utilise some clustering algorithms to detect for outliers. DBScan (5) is one of my favourites. By tweaking the min sample (for a cluster) and epsilon parameters, you can make the algorithm assume a single cluster and highlight any outliers as "noise observations".
Bear in mind that DBScan takes a long time to run and in terms of computation time it may struggle with 10M+ observations (still give it a go). The other algorithms should not have a problem as long as the data fits in your memory (no big data solution here).
Another general note is that, for all of the above, you will need to enter an estimated "outlier fraction" as a model input. So none of these is actually fully "unsupervised". 
With regards to including missing data as outliers, not sure if this makes sense or whether it is practically possible. Your remark on this is not clear to me.
Finally, this link should give you a good start in terms of how to use some of these novelty detection algorithms, already implemented in Python. Also, the implementation for DB-Scan is here.


*

*(1) Rousseeuw, P.J., Van Driessen, K. “A fast algorithm for the
minimum covariance determinant estimator” Technometrics 41(3), 212
(1999) 

*(2) http://www.jmlr.org/papers/volume2/manevitz01a/manevitz01a.pdf 

*(3) Liu, Fei Tony, Ting, Kai Ming and Zhou, Zhi-Hua. “Isolation forest.”
Data Mining, 2008. ICDM‘08. Eighth IEEE International Conference on.

*(4) Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000,
May). LOF: identifying density-based local outliers. In ACM sigmod
record.

*(5) Ester, M., H. P. Kriegel, J. Sander, and X. Xu, “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise”. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland, OR, AAAI Press, pp. 226-231. 1996

