Anomaly detection in time-series with categorical data

Question

There are many tutorials/packages in Python to detect anomalies in time-series given that the time-series is numerical.

Currently, I have a time-series that is categorical, i.e. the time-series data said that, at time XXX the event AAA occurred. I want to detect anomalies for this data. For instance, if too many events BBB occurred in a short period of time...

Where should I start, especially in Python?

Answer 1

What you have there seems to be an uneven/unequally/irregularly spaced time series, consisting of discrete (in time) events.

The most straightforward approach for anomalies in number-of-events would be to transform the data into a regular time-series, and use a general-purpose anomaly detection model.

import pandas

events = pandas.DataFrame.from_records([

# B normally happens once per day
{ 'time': '2023-03-22T18:22:00', 'event': 'B' },
{ 'time': '2023-03-23T18:22:00', 'event': 'B' },
{ 'time': '2023-03-24T18:22:00', 'event': 'B' },

# skipped a day
#{ 'time': '2023-03-25T18:22', 'event': 'B' },

# many times in one day
{ 'time': '2023-03-26T16:22:00', 'event': 'B' },
{ 'time': '2023-03-26T18:22:00', 'event': 'B' },
{ 'time': '2023-03-26T20:22:00', 'event': 'B' },

# A once per week
{ 'time': '2023-03-26T20:22:00', 'event': 'A' },

])
events['time'] = pandas.to_datetime(events['time'])

print(events)

# Transform to regular time-series with counts
time_bins = '1d'
regular = events.set_index('time').groupby('event').resample(time_bins).count().rename(columns={'event': 'count'})
print(regular)

# Transform into one column per event
data = regular.reset_index().pivot(index='time', columns='event', values='count').add_suffix('_count')
data = data.fillna(0.0)
print(data)

# Can do further feature engineering here, like split time into weekday/time-of-day
# and then pass to a standard Anomaly Detection method, such as IsolationForest

Should print

                 time event
0 2023-03-22 18:22:00     B
1 2023-03-23 18:22:00     B
2 2023-03-24 18:22:00     B
3 2023-03-26 16:22:00     B
4 2023-03-26 18:22:00     B
5 2023-03-26 20:22:00     B
6 2023-03-26 20:22:00     A

                  count
event time             
A     2023-03-26      1
B     2023-03-22      1
      2023-03-23      1
      2023-03-24      1
      2023-03-25      0
      2023-03-26      3

event       A_count  B_count
time                        
2023-03-22      0.0      1.0
2023-03-23      0.0      1.0
2023-03-24      0.0      1.0
2023-03-25      0.0      0.0
2023-03-26      1.0      3.0