I have built a prototype Naïve Bayes Classifier in an Excel spreadsheet. My data is a transaction (an order) with 13 parameters. This translates directly to a feature vector of (feature_1, feature_2, …, feature_13). Each parameter of an order, if it is incorrectly set, can be the cause of an error in the order. I am trying to classify each order according to the cause of the error in the order. So I’m modelling 14 classes: each of the parameters, plus an ‘other’ class for some unknown error that doesn’t fit into any of the other classes.
My question is: how do I measure the increase in accuracy of the system after it is trained with each new order? And on a related note, should I expect the average increase in accuracy (after each new order) to drop over time? I.e., the system will get less and less value out of each new feature vector that is fed into it?