0
$\begingroup$

I am studying a dataset of time series for different users. The dataset contains records of actions (or registrations) of the users over time. I have data of a whole week for about 80,000 users.

Objective

At the moment, I'm concerned about the count of registrations over time. My objective right now is to study the 'patterns of registrations' of different users. I'm interested in cluster them according to their registration pattern and detect 'anomalous' user in terms of their registration behavior, since it is of our interest to identify group of users and also users that are considerably apart from groups of other users.

Considerations

The registrations made by a user during a week are not regular or continuous. A user can perform registrations every day while other may exist just for one day.

Registrations during the day may vary. Actually I'm interested on identify different behaviours during the day.

My current approach

I am thinking in different approaches. I'm studying the daily variation of the registrations to identify patterns in terms of registrations per hour. My idea was to define for each user an array with the number of registrations per hour registered during the week and normalize it, since not all users are present everyday. I can define this 'distribution' for each user.

Here I encountered many problems

  • In the first place, if I get the overall distribution per hour considering all users and then I compare the distribution of each user against this to determine if they are anomalies I think it might not be the right approach. This is because: 1) I doubt about the 'statistical correctness' of performing this operation, although I've reading about statistical distance to compare distributions. 2) I think this will not allow me to cluster different daily behaviors (e.g., users that register more during the night than during the morning and viceversa)

  • I'm not sure how can I determine this 'similar behavior' between users. Maybe some of them have uniform registrations during the day while other exhibit peaks at different times.

My main doubt is how can I cluster users based on their daily activity. I think if I manage to do clustering the anomaly detection will be derived from that methodology.

I'm not sure where I should start to approach to this problem more systematically, so I appreciate any insights.

Best regards,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Your approach of using the overall distribution is simple but sounds statistically correct. It is a good start. There are variations to it, such as simply declaring everything in the lower/upper x% as outliers; using interquartile ranges like in boxplots to identify outliers, or even using Gaussian mixture models to identify clusters within the distribution.

The choice of the method highly depends on what you understand under the 'patterns of registrations' and 'behavior'. If you can quantify it, then you can compute a distance or similarity matrix and use the matrix within some clustering algorithm. For example, organize the data as the number of registrations per user (columns) per each hour (rows) of the week. Then, a matrix of pairwise correlations between each pair of columns will be a similarity matrix. Alternatively, compute pairwise Euclidean distances which can be arranged in a distance matrix.

More methods including model-based approaches can be found in this book by Maharaj et al. (2019).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Regarding your second paragraph, an example. Considering all the registers per hour I see a 'continuous' with top values during working hours and lower values during the night. I'd like to identify if there are users that exhibit different type of behaviours in terms of registrations vs. hour curve. For example, users with registration peaks during the night or users that present an impulse train-like distribution. I'd like to cluster the different shapes. What you propose in the second paragraph seems like an interesting approach to do this, I'll explore it. $\endgroup$
    – tms
    Commented Sep 12, 2023 at 14:16
  • $\begingroup$ Glad to help. If you will be calculating distances, consider rescaling the data before calculating the distances. It will allow to focus on the patterns rather than the differences of means. For example, you can rescale each time series by converting it to z-scores (subtract its mean and divide by standard deviation), or to some interval like [0, 1]. $\endgroup$
    – Slava
    Commented Sep 13, 2023 at 15:27
  • $\begingroup$ Hi, I've explored this solution, however, I'm facing two issues. - Using IQR to filter outliers doesn't seem to fit the problem since the count data is highly concentrated on low values (80% is 2); hence, everything different from the median is consider an outlier. Outliers seems interesting so I wouldn't like to discard all of them - I have some doubts about the similarity matrix. If consider many user the matrix seems not to be scalable. If I have for example 80,000 users the table will be a 80,000 by 80,000 matrix. Could you give me any comments about these issues? $\endgroup$
    – tms
    Commented Sep 27, 2023 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.