0
$\begingroup$

I am applying an unsupervised learning algorithm for building an anomaly detection using OneClass SVM method and then plotted it to visualize how it looks.

I got 2 clusters: one red and the other blue. The red cluster corresponding to 1 (not an anomaly) and the blue cluster having a value of -1 (anomaly).

What I would like is to obtain the exact value at which both the clusters get separated.

My Code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing

data = pd.read_excel('gpmd.xlsx', header = 0)
X = data.loc[:, ['ContextID','BacksGas_Flow_sccm']]

min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X.values[:,[1]])

from sklearn.svm import OneClassSVM

ocsvm = OneClassSVM(nu = 0.05, kernel = 'rbf', gamma = 'scale')
y_ocsvm1 = ocsvm.fit_predict(X_minmax[:,[0]])

plt.scatter(X.values[y_ocsvm1 == 1, 0], X_minmax[y_ocsvm1 == 1, 0], c = 'red', label = 'cluster1')
plt.scatter(X.values[y_ocsvm1 == -1, 0], X_minmax[y_ocsvm1 == -1, 0], c = 'blue', label = 'cluster2')
plt.ticklabel_format(useOffset=False)
plt.yticks(np.arange(min(X_minmax[:,[0]]), max(X_minmax[:,[0]]), 0.03))
plt.legend()
plt.show()

The graph I obtained after running this code is:

scatter plot

From just looking at the picture, we can roughly say that the red cluster is separated from the blue cluster at a value between 0.72 and 0.75. I would like to know if there is a way to know the exact value where these two clusters get separated.

Edit 1

In the picture below we can see that a green line separates both the clusters(roughly) and I would kind of like to find such a line and its corresponding y-axis value Picture 2

Edit 1: My dataset looks like this:

ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
7289973 09:12:48.502    49.56054688 1.953125
7289973 09:12:48.603    49.56054688 2.05078125
7289973 09:12:48.934    99.85351563 2.05078125
7289973 09:12:49.924    351.3183594 2.05078125
7289973 09:12:50.924    382.8125    1.953125
7289973 09:12:51.924    382.8125    1.7578125
7289973 09:12:52.934    382.8125    1.7578125
7289999 09:15:36.434    50.04882813 1.7578125
7289999 09:15:36.654    50.04882813 1.7578125
7289999 09:15:36.820    50.04882813 1.66015625
7289999 09:15:37.904    333.2519531 1.85546875
7289999 09:15:38.924    377.1972656 1.953125
7289999 09:15:39.994    377.1972656 1.7578125
7289999 09:15:41.94     388.671875  1.85546875
7289999 09:15:42.136    388.671875  1.85546875
7290025 09:18:00.429    381.5917969 1.85546875
7290025 09:18:01.448    381.5917969 1.85546875
7290025 09:18:02.488    381.5917969 1.953125
7290025 09:18:03.549    381.5917969 14.453125
7290025 09:18:04.589    381.5917969 46.77734375

What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The ContextID is more like a product number.

$\endgroup$
  • $\begingroup$ I don't know yore data, but from the plot it does not look like the algorithm had found any meaningful separation. $\endgroup$ – Tim Apr 2 at 12:23
  • $\begingroup$ Hey @Tim, why do you say that the algorithm has not found any meaningful separation? I would really be interested in knowing $\endgroup$ – Junkrat Apr 2 at 12:27
  • $\begingroup$ I have also edited the question and added a small sample of how my dataset looks like $\endgroup$ – Junkrat Apr 2 at 12:29
  • 1
    $\begingroup$ Visually. I guess, if you didn't color the "classes", you wouldn't draw the line by yourself in this place. The classification on top is strange, but on the bottom it seems to choose inliers. Bus as I said, I don't know your data. I'm just saying that the classification & what you've shown on the plot, do not seem very related to each other. $\endgroup$ – Tim Apr 2 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.