There is a somewhat off-the-beaten-path supervised document classification problem I am trying to tackle. For the sake of simplicity, let's say we are using the bag-of-words approach. Usually, we would feed the feature vectors (in this case, word vectors) into a classifier such as SVM. However, in the data, we have additional human input that indicates certain words are strongly indicative of a particular class. There is a list of such words. For example:

  • abc indicates a 75% chance of the document being in class A
  • xyz indicates a 80% chance of being in class B
  • pqr indicates a 70% chance of being in class A

We don't know exactly how this list was generated, however, and cannot take the phrase "x% chance" to denote actual probability. Also, the list is short, only about 20 words per class. In particular, this means that most words do not have an a priori bias towards a specific class.

In a standard SVM classification process, incorporating such prior knowledge of feature weights is not possible, to the best of my knowledge. Is there an algorithm that can be used for supervised classification (since we have class labels for a large number of these documents) but instead of purely learning the word weights during training, can also incorporate a predefined set of words that have been indicated as more important for a class label?

In more formal terminology: we have a prior distribution on the features, and labeled documents, and we want to use these two together for classification.

Note: I am looking for an approach that will yield explicit feature weights in order to make the AI explainable, so multilayer perceptrons are not an option.

  • $\begingroup$ You could add more weight to the known good features for each class during training, i.e. use label-specific feature weights. If you know that these terms are highly discriminatory with respect to classification, you could also experiment with using just a single set of feature weights which emphasize all the terms. This general approach should work with all types of classifiers. $\endgroup$ Jul 13, 2018 at 12:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.