0
$\begingroup$

I have a dataset consisting of numbers representing the values of a KPI (Key performance indicator) collected over a period of time. I would like to implement an algorithm to classify the data as normal or anomaly (and I am planning to use logistic regression or maybe decision trees).

However, the problem is that I don't know which values are normal which values are problematic. So what should I do beforehand to create a training set for my classification algorithm ? Should I use maybe clustering algorithms ?

$\endgroup$

2 Answers 2

1
$\begingroup$

If you don't have labeled data (telling whether it belongs to class "normal" or "anamoly"), you will have to use a cluster analysis - for grouping the objects.

Logistic regression is a supervised learning algorithm and requires you to have data that has classes. Same goes for decision trees.

Look at this with respect to validating unsupervised machine learning algorithms.

$\endgroup$
0
$\begingroup$

You can check other threads tagged for multiple examples. More examples can be found in this thread about finding outliers in multivariate data as this is a closely related topic. There exist specialized algorithms for dealing with such cases. One-class SVM is one of such algorithms. It is nicely described Microsoft Azure documentation:

Typically, the SVM algorithm is given a set of training examples labeled as belonging to one of two classes. The SVM algorithm represents the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to one category or another, based on which side of the gap they fall on.

... one-class SVM, the support vector model is trained on data that has only one class, which is the “normal” class. It infers the properties of normal cases and from these properties can predict which examples are unlike the normal examples. This is useful for anomaly detection because the scarcity of training examples is what defines anomalies: that is, typically there are very few examples of the network intrusion, fraud, or other anomalous behavior.

So the basic idea is that you use your data to learn about what is "normal", or "typical" and then classify cases that are distant from what is normal as anomalous. This idea can be easily extended beyond the specialized anomaly detection algorithms.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.