I am building a natural language understanding component for a dialog system. The input is a natural language sentence (usually a short one), and the output should be a set of zero or more classes from a pre-specified set. For example:
input: "I offer a salary of 20000 per month for work as a programmer with a company car" output: ["OFFER(Salary=20000)", "OFFER(Job=programmer)", "OFFER(Car=yes)"]
I collected some examples from real dialogs, in order to train a classifier. My current approach is:
- Features: I take the words of each sentence as features (I also tried word bi-grams and letter n-grams but the performance was worse).
- Training: I train a separate binary classifier for each class. As positive examples, I take all sentences that have this class, and as negative examples, I take all sentences that don't have this class.
- Running: I run each of the binary classifiers on the new sample, and return the set of classes that correspond to the classifiers that answered 'yes'.
This approach lead to too many false positives. Since my dataset contains many sentences with many classes (about 6 classes per sentence), the classifier gets confused between words that belong to different classes. For example, it takes the words "work as a programmer" as positive signal to the class "OFFER(Salary=20000)", because they appeared together in many training samples.
So, I edited my training data and separated each sample sentence to several sub-sentences, such that each sub-sentence matches just one class:
input: "I offer a salary of 20000 per month" output: ["OFFER(Salary=20000)"] input: "for work as a programmer" output: ["OFFER(Job=programmer)"] input: "with a company car" output: ["OFFER(Car=yes)"]
The new approach lead to too many false negatives: since every sentence is a negative example of all classes except its own class, the classifier takes the words "a salary of 20000" as negative signal to the class "OFFER(Job=programmer)". Therefore, for compound sentences such as "I offer a salary of 20000 per month for work as a programmer with a company car", the classifier finds a single class and misses all other classes.
I also created a mixed dataset - half of the one-class samples and half of the multi-class samples (the other half I used for testing), and got much better result. However, this approach seems like a hack, and I would like to know if there is a more professional approach.
Any suggestions for solution will be welcome...
NOTE: I use the winnow classification algorithm, which currently has the best score (I also tried Naive Bayes, SVM, neural network and perceptron).
- Performance of classifier trained on multi-class sentences and tested on single-class sentences: Precision=69% Recall=97% F1=81%
- Performance of classifier trained on single-class sentences and tested on multi-class sentences: Precision=99% Recall=32% F1=48%
- Performance of classifier trained on a mixed dataset (1/2 single-class and 1/2 multi-class), and tested on the other half: Precision=88% Recall=89% F1=89%