I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and I am currently trying to figure out which algorithm or algorithms are useful in solving time series anomalies. Basically I do know I have to implement a supervised model because I have training data to train my model, and I kind of know what would an anomaly (outlier) be, considered on the threshold.

The main project consists on analyzing daily files and detecting fluctuating changes in some of the records and reporting them as possible anomalies (if they are considered one based on the model). The files are generated at the end of a day and mi program needs to check them on the morning of the next day to see if there is an anomaly. However, I need to check anomalies file vs file, NOT within the file. This means that I have to compare the data of every file and see if it fits to the model I would create following the specific algorithm. What I'm trying to say is that I have some valid data that I will apply the algorithm to in order to train my model. Then I have to apply this same model to other files of the same format but, obviously, different data. I'm not looking for a prediction column but rather detecting anomalies in these other files. If there is an anomaly the program should tell me which row/column has the anomaly and then I have to program it to send an email saying that there is a possible anomaly in the specific file.

Like I said I am new to machine learning and I would rather focus on learning the best algorithm (or algorithms) that can solve time series anomaly detection instead of studying ALL of the algorithms since I have a short time to solve this. I heard algorithms like neural networks or k-means, but I don't know which one applies best to my case.

I appreciate your help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.