I want to build a system, that can classify known classes in a supervised way and at the same time tells if there is a new anomaly class it has not seen before. The user can then label that unknown class (as normal or a certain failure class) so it will be detected next time.

I can imagine two approaches:

A) 1 classification model - a high uncertainty / low confidence would indicate an unknown class

B) 1 classification model + 1 anomaly detection system (combine their outputs 'somehow', not sure what would be the best way).

Btw. it is about audio data.

Any idea or resources what is a good or best practise approach for this (I think common) requirement?

  • $\begingroup$ Say that you build it, initially it predicts two classes and marks one case as anomaly--if you train your classifier on single example it would badly overfit... What if on next iteration another case would be marked as anomaly--would you train for another single-observation class? $\endgroup$ – Tim Mar 31 '18 at 10:21
  • $\begingroup$ Do you have well-defined idea what an outlier will be? I think you are right to suspect that a 1+1 approach is better. That said use domain knowledge. Eg. If as you say these are sound data, and you expect speech utterances, check what the $F_0$ (fundamental frequency) of the recording are. If they are something rather outside norm (eg. $F_0$ ~50H i.e. a baritone talking out of a cave) probably they are outliers... $\endgroup$ – usεr11852 says Reinstate Monic Mar 31 '18 at 11:51
  • $\begingroup$ Yes it is about audio data, and it should detect failures based on anomalous noises. A challenge is that the same rare event which is not a failure would be always detected as anomaly. Another rare event could be a real anomaly. So I need some classification that can over-rule the anomaly detection. $\endgroup$ – MikeHuber Mar 31 '18 at 12:01
  • $\begingroup$ Do you have an idea what is an "anomalous noise"? If yes, built a sample of "anomalous noises" and use them as third class. $\endgroup$ – usεr11852 says Reinstate Monic Mar 31 '18 at 12:10
  • $\begingroup$ No unfortunately not at the beginning. The idea is is to label the anomalous noises when they occur the first time, so that they will be classified next time. I could think of data augmentation to reduce overfitting a bit after the first examples. $\endgroup$ – MikeHuber Mar 31 '18 at 12:20

The case where you want to detect that a certain input is not represented in the dataset, is sometimes referred to as "novelty detection", a slight variation on anomaly detection.

Your suggested approach A) is the simplest one (single model), so try that first.

If it does not work OK you could try a cascade of NoveltyDetector -> Classifier. But the performance in the detector can easily become a limitation, as well as disagreement between the detector and classifier. You can see the choice of architecture as a hyperparameter, and approach it in the standard supervised learning way: Use a held-out testset and cross-validation in order to determine what works best for your dataset.

A good starting point is to have a solid feature representation. An audio embedding is a compact representation of a short period of audio (few seconds). The AudioSet project has released strong pretrained CNN models called VGGish that can be used to produce such embeddings. Github. These are 128 dimensional with on orthogonal basis, which makes it easily to build machine learning on top using simple methods and very small datasets. Linear models, kNN and tree-based methods should all do pretty well, both for classification and novelty detection.

The user can then label that unknown class...so it will be detected next time.

Learning from a single sample (one-shot learning) is very challenging, as is automatically updating the model (online learning). For that reason it is not commonly done these days, and best practice is scarce.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.