Unsupervised Outliers detection on time series

Question

So I am looking ways to improve my current implementation of detecting outliers in work schedule. My data set is badge swipes for people. The current implementation finds outliers on in-times and out-times separately using simple standard deviation approach. The problem with this approach is,

1. We have to manually find in-times and out-times. This is tricky for people working overnight.
2. Since the outlier detection involves mean, in cases of changed schedule or rotating shift the mean way off.

Here is the sample of data;

      PersonID            event_ts
    0   104756 2017-02-13 10:37:29
    1    38469 2017-05-10 09:15:11
    2   111130 2017-05-05 12:08:07
    3   601398 2017-04-05 18:14:10
    4    33945 2017-02-08 09:10:31
    5   101294 2017-05-17 12:28:51
    6    39476 2017-06-23 16:01:44
    7    31791 2017-03-15 18:42:12
    8   114090 2017-03-14 20:33:27
    9    33380 2017-02-22 16:03:22

Now, another challenge with this data is that training data will be contaminated with outliers. And it's very difficult to clean because of the number of people in the dataset and because of the changing or rotating shifts.

Now, I tried used GaussianMixter model, and adapted the dates to ordinal for the algorithm use. And instead of a complete timestamp, I am using hourly data with IS_SWIPE flag for at least one swipe in that hour. So that data looks something like this.

                                     PersonID  Date_ordinal  hour  IS_SWIPE
    0  0x002935373242324333352D303533332D3443        736384     0         0
    1  0x002935373242324333352D303533332D3443        736384     1         0
    2  0x002935373242324333352D303533332D3443        736384     2         0
    3  0x002935373242324333352D303533332D3443        736384     3         0
    4  0x002935373242324333352D303533332D3443        736384     4         0
    5  0x002935373242324333352D303533332D3443        736384     5         0
    6  0x002935373242324333352D303533332D3443        736384     6         0
    7  0x002935373242324333352D303533332D3443        736384     7         0
    8  0x002935373242324333352D303533332D3443        736384     8         1
    9  0x002935373242324333352D303533332D3443        736384     9         0

Here is my general code that I am trying to work with;

    df = pd.read_sql_query("""SQL Query""", cnxn, 
             parse_dates=['Date','Datetime'])
    df['Date_ordinal']=df['Date'].apply(datetime.toordinal)
    df = df.sort_values(by=['Datetime'])
    train_set, test_set = train_test_split(df[['Date_ordinal','hour','IS_SWIPE']], test_size=0.4)
    A = mixture.GaussianMixture(n_components=3, covariance_type='tied').fit(train_set)
    print A.predict(test_set)[:10]

> [1 0 2 0 2 0 2 2 2 1]

Now, where I am stuck is,

1. I don't know how to interpret the output of the predict function. What does value 0, 1 and 2 mean, I checked the Scikit-learn documentation here but it only talks what the function does nothing about the output. Also, they don't seem to be in the same order as input and keep changing the order every time the function is called.
2. Did the fit function take into account the outliers in the train set, most likely not, right?
3. If so I was thinking of using the "Robust covariance estimation and Mahalanobis distances relevance" method but I am not completely sure if I understand it and how would I adapt my code to it?
4. Would the Mahalanobis Algorithm work with the raw timestamp with changing or rotating work schedule? Or do I still have to use the modified data as above?

Any help is much appreciated. And as you can see I new to Data Science and still learning, so please excuse me for my ignorance.

Answer 1

1. the numbers are the identifiers to which class each row belongs. You indicated n_components = 3, so your classes are 0, 1, 2. If you had indicated n_components = 4, your results would have been 0, 1, 2, 3. When you fit, the exact number you get for your first row doesn't really matter, as long as the rows with the same numbers still share the same numbers, i.e. on attempt 1 you get [0, 2, 0] for rows [0, 1, 2] on attempt 2 you may get [2, 1, 2] for the same rows, but as long as row 2 and row 0 have the same value it's ok

2. GMM is clustering your rows into gaussian distributions. It will set a label to all points. I don't understand your definition of an outlier. Is it for times in/out that do not make sense? e.g. HR swiping in at 3 am

3. go through the example on sklearn

4. you will need numeric data. Stick with your modified dataset