I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered to be an unsupervised learning, most do not require labeled data as mentioned here. Recently I re-faced to following fundamental questions and literature review in this regard to updating myself and find clear answers if I want to use DL models, e.g., Deep Autoencoder:
- Deep Learning (DL) is supervised or unsupervised?
- Does splitting data make sense in unsupervised settings?
- How can we calculate the loss function (MSE, MAE, etc.) for the unsupervised setting? Is it common?
using python & sklearn:
from sklearn.metrics import mean_squared_error #Feature Selection criterion = mean_squared_error(y, predictions)
import numpy as np criterion = np.mean((y_test - est.predict(X_test))**2)
Personally, I'm targeting Clustering methods, and its validation is based on similarities as well as isolation forest and move on Autoencoders for DL as recommended here. Of course, I'm aware that Cluster analysis is not something to automate fully, and It is an explorative method. On the other hand, not completely reliable I was always imagining the main reason for distinguishing between supervised and unsupervised is the existence of label features in data. My picture about DL is mostly the high number of hidden layers and leave the responsibility of feature engineering for DL models. I do like this vision concerning comparing unsupervised anomaly models. I believe that we need still to splitting data and calculate loss function despite unlabeled data due to the fact that:
- calculating the metrics on the training set would likely lead to overfitting, and then we need the testing set to evaluate the model. Generally speaking, to track the level of overfitting while we are experimenting with our network.
- get an estimate of how many epochs we should train for
Additionally, I read this online article Issues with Unsupervised Learning and Why still we need them.
I'm kind of lost it seems; on the one hand, we can't evaluate the unsupervised method on unlabeled data. However, on the other hand, It is still possible to split data apply Clustering On Informative Features or use Cross-validation (CV) nevertheless result wasn't promising. reference
I'm not sure the new generation of clustering methods like based on the Correlation Clustering functional (CC) or Cromatic Correlation Clustering (CCC) or one-class models or Positive Unlabeled (PU) learning could be good candidates. I wonder if it is worth it to benefit from Active Learning to improve the classifier in this concept since human intervention can't be imagined, which is a special form of semi-supervised learning. Adversarial learning offered by Ian Goodfellow and Nicolas Papernot can be interesting as well.
I know the above-mentioned questions look naive, but any update or clarification, even in the short form answers for this trilogy questions, will be highly appreciated and help me get rid of this confusion and shape my vision correctly.