What is the-state-of-art for unsupervised Anomaly models through unlabeled data regarding evaluation/validation in 2020? [closed]

Question

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered to be an unsupervised learning, most do not require labeled data as mentioned here. Recently I re-faced to following fundamental questions and literature review in this regard to updating myself and find clear answers if I want to use DL models, e.g., Deep Autoencoder:

• Deep Learning (DL) is supervised or unsupervised?
• Does splitting data make sense in unsupervised settings?
• How can we calculate the loss function (MSE, MAE, etc.) for the unsupervised setting? Is it common?

using python & sklearn:

from sklearn.metrics import mean_squared_error
#Feature Selection
criterion = mean_squared_error(y, predictions)

or numpy:

import numpy as np
criterion = np.mean((y_test - est.predict(X_test))**2)

Personally, I'm targeting Clustering methods, and its validation is based on similarities as well as isolation forest and move on Autoencoders for DL as recommended here. Of course, I'm aware that Cluster analysis is not something to automate fully, and It is an explorative method. On the other hand, not completely reliable I was always imagining the main reason for distinguishing between supervised and unsupervised is the existence of label features in data. My picture about DL is mostly the high number of hidden layers and leave the responsibility of feature engineering for DL models. I do like this vision concerning comparing unsupervised anomaly models. I believe that we need still to splitting data and calculate loss function despite unlabeled data due to the fact that:

1. calculating the metrics on the training set would likely lead to overfitting, and then we need the testing set to evaluate the model. Generally speaking, to track the level of overfitting while we are experimenting with our network.
2. get an estimate of how many epochs we should train for

Additionally, I read this online article Issues with Unsupervised Learning and Why still we need them.

I'm kind of lost it seems; on the one hand, we can't evaluate the unsupervised method on unlabeled data. However, on the other hand, It is still possible to split data apply Clustering On Informative Features or use Cross-validation (CV) nevertheless result wasn't promising. reference

I'm not sure the new generation of clustering methods like based on the Correlation Clustering functional (CC) or Cromatic Correlation Clustering (CCC) or one-class models or Positive Unlabeled (PU) learning could be good candidates. I wonder if it is worth it to benefit from Active Learning to improve the classifier in this concept since human intervention can't be imagined, which is a special form of semi-supervised learning. Adversarial learning offered by Ian Goodfellow and Nicolas Papernot can be interesting as well.

I know the above-mentioned questions look naive, but any update or clarification, even in the short form answers for this trilogy questions, will be highly appreciated and help me get rid of this confusion and shape my vision correctly.

Answer 1

Is DL supervised or unsupervised?

Although DL is usually applied to supervised learning (classification and regression), it can be used to unsupervised learning as well. You can read about it in this review. You mentioned the autoencoder in your question, and it is a nice example of unsupervised DL: it does not attempt to predict a target variable, it only seeks to find a low dimensional representation for your data, for purposes of dimensionality reduction, denoising, clustering, etc.

Does splitting data make sense in unsupervised settings?

I am not sure here. This answer on Stack Overflow discusses this a bit, but I did not find it very convincing. If you have some instances which you know to be anomalous, you could use these labels to test your model (in a kinda semisupervised way, since your model is unsupervised but the model evaluation is supervised). Even if you don't have labeled data, perhaps it would be helpful to check how many instances your model considers anomalous in a test/validation dataset (in case of overfitting, one would expect that your model considers too many test instances to be anomalous).

How can we calculate the loss function (MSE, MAE, etc.) for the unsupervised setting? Is it common?

If you are doing cluster analysis, perhaps you should focus on different metrics/loss functions such as the ones provided by Scikit-learn for clutering evaluation. For dimensionality reduction methods such as PCA and autoencoders, it makes sense to use metrics like MSE. If we denote by $f$ the function performing the dimensionality reduction and $g$ the function mapping the encoded representations back to the reconstructed instances, you could have something like: $$\text{MAE}=\frac 1n \sum_{i=1}^n ||x_i-g(f(x_i))||^2$$

Also, make sure you have a clear image of the question you want to answer. Clustering and anomaly detection algorithms will give you different information, and it's important to decide which is better for you. It seems that, in your application, the goal is to flag unusual traffic data. For this particular task, anomaly is a better fit than clustering. [Some algorithms, like DBscan or even GMM, can do both clustering and anomaly detection. Still, you have to be aware of the difference and the importance of each task in your application.]

Hope it was helpful!