How do I compare short alphanumerical text and classify the outcome with Machine Learning?

Question

I have an adjudication automation problem.

I have many lists of different instruments, each with a set of specifications. These specifications were sent to vendors that tender on the instruments. I receive back information from the vendors specifying what they can provide. I now need to compare the requirement with what they can supply and label the comparison of each specification with a "C" for compliance, "NC" for non-compliance, "PC" for partially compliant, "N/A" for not applicable and "INA" for information not available.

The judgement cannot be made with a simple comparison as the judgement may depend on some of the other specification fields of the instrument. The specifications are mostly text, but sometimes numbers as well.

Here is an example of a table with some data: Limmited Data Example

I want to transform the text into features, but I have difficulty in determining the best route:

1. The one way that I want to proceed is to vectorize the strings by tokenizing and a bag of words, but I do not know if this will generalise well because of the comparative nature of the test. So I thought I would then create 3 bags of words; one for the spec value, one for the vendor-value-and one for the line number(which is alphanumerical) and stack the vectors together as a feature vector
2. The second way I am considering is doing several similarity tests between the specified value and the vendor provided value and use the outcomes of the similarity tests as features for training.

How should I typically start with such a classification?

I want to complete this problem in C#

Answer 1

I would suggest the first approach you described. That's what I was thinking I would do before getting to your suggestions. I would keep the different fields separate. Build a bag-of-words for each of them and then make that your feature space. That is a very good start I think. recognize however that this will cause a very sparse matrix, not all the words will appear in all the examples. Therefore, I would pass some sort of PCA or LDA to get rid of features that will not contain any useful information, these are the words that appear in every single example ('kPag'). I don't think the unit would give you much information about its compliance. So it should be rejected. Make sure to use a shallow machine learning model, don't try something to fancy because it will over-fit very quickly, 1 due to the small dataset I assume this problem will have and also because of the sparsity of the feature matrix.

Answer 2

Text mining via N-grams would be perfect for this. It will start by looking at all count frequencies of singletons: a,b,c,d..., then doublets: ab,ac,ad,ae,...,triplets, aab,aac,aad,aae,.... up to what ever you want. If you go above 5-grams the algorithm can appear to hang since there are so many combinations(permutations).

This falls in machine learning because the algorithms behind n-gram analysis are statistical and machine learning based, such as spherical k-means clustering.