currently I am working on a project for my final exam. The data is coming from a streaming plattform. The data I am using are some logging data (data when customers have problems with the streaming service). My knowledge in this topic is not the best but I will try to explain my current progress and what my task is. In the first steps I made some data preparation. After this step I have following data structure:
- eventTime - time when error occured
- deviceId - device identifier
- errorCode - specific code, which can be used to get more information of the error
So these are the only attributes I have. All other attributes, which are not listed, were irrelevant. In the first step I created time ranges (15 minutes). For each time period I counted the errors. Now I want to make some time series analysis. I need some algorithms/strategies to detect:
- jumps (e.g. count was static 5, but suddenly jumped to 10 and it stays at 10)
- peaks/anomalies (static 5, suddenly 15 for on range, then back to 5
I found a moving average method. But with this the data is slightly manipulated and it is only suitable to see if there are trends. Do you have some suggestions which methods I could use for each of the points? One more problem is that I need to do a analysis for every error type (there are about 20000). Looking at each graph would take too much time. For example: I want the top 30 errors which have an upgoing trend. Is there a solution for this? It would be helpful if a algorithm just tells me wether the error makes problems or not.
It would be nice if someone could help me with my problem. Thanks in advance!