I have a dataset of text-documents, which are all considered valid. Also I have a set of manually defined features that are required to be in the document for it to be considered valid. The features can be represented by different words(synonyms).

I want to test new documents in how much they adhere to the features. It is not a binary classification problem, as I don't have examples of invalid documents. Furthermore, the documents that are not valid will most likely be incomplete, so they will lack some. The result should be continuous and point to the feature that are missing. What should I use?

  • 1
    $\begingroup$ This sounds like an anomaly-detection problem. $\endgroup$ – Tim Aug 4 '20 at 13:53
  • 1
    $\begingroup$ Can you elaborate more on what the features are? Are they just specific word counts? $\endgroup$ – Skander H. Aug 5 '20 at 19:27
  • $\begingroup$ Yes they are word counts. $\endgroup$ – Borut Flis Aug 5 '20 at 20:39
  • 1
    $\begingroup$ So let me see if I understand: You have a set of features which consist of word counts of groups of related relevant terms. A document is valid if those word counts within a certain range, and invalid if the word counts fall outside of it. If this is correct, why do you need a complex algorithm ML or otherwise? Why can't you just set up a bunch of if-then rules for each feature to test for validity? Especially since you don't just want to detect an invalid document but also detectl which feature is faulty? $\endgroup$ – Skander H. Aug 6 '20 at 21:48
  • $\begingroup$ The problem is more probabilistic, it can not be written as a if-then rule. Anyway I am using one-class SVM now. $\endgroup$ – Borut Flis Aug 7 '20 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.