currently I am working on a project for my final exam. The data is coming from a streaming plattform. The data I am using are some logging data (data when customers have problems with the streaming service). My knowledge in this topic is not the best but I will try to explain my current progress and what my task is. In the first steps I made some data preparation. After this step I have following data structure:

  • eventTime - time when error occured
  • deviceId - device identifier
  • errorCode - specific code, which can be used to get more information of the error

So these are the only attributes I have. All other attributes, which are not listed, were irrelevant. In the first step I created time ranges (15 minutes). For each time period I counted the errors. Now I want to make some time series analysis. I need some algorithms/strategies to detect:

  • trends
  • jumps (e.g. count was static 5, but suddenly jumped to 10 and it stays at 10)
  • peaks/anomalies (static 5, suddenly 15 for on range, then back to 5

I found a moving average method. But with this the data is slightly manipulated and it is only suitable to see if there are trends. Do you have some suggestions which methods I could use for each of the points? One more problem is that I need to do a analysis for every error type (there are about 20000). Looking at each graph would take too much time. For example: I want the top 30 errors which have an upgoing trend. Is there a solution for this? It would be helpful if a algorithm just tells me wether the error makes problems or not.

It would be nice if someone could help me with my problem. Thanks in advance!


The algorithm you needs is change point detection.
It's a ton of ideas.
In python, ruptures is a package, it's really nice to understand cp detection.
Basic idea is separated 3 component.

1.How to define searching area.
(all data at same time, window..)

2.How to compare the data.
(model, statistic.....)

3.How to restrict cp.
(by increasing cp, model is easy to fit)

This theory can be programmed manually easily, so try to understand algorithm and try it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.