Optimizing across multiple classification models (different training sets)

Question

I have a large dataset consisting of many features for each record which includes either a GOOD or BAD outcome. Three metrics have been created for classification purposes using three respective Bayesian Belief Networks:

• Classification Network including outcome variable trained on both GOOD and BAD records
• GOOD Network trained on only GOOD records
• BAD Network trained on only BAD records

For the GOOD and BAD Networks, a most likely state is predicted for each feature given information on all other features and compared with the actual value present. The GOOD and BAD metrics are then defined by the percent of features correctly predicted.

QUESTION

Are there any standard methods that exist for defining optimal thresholds and rule sets for classification on separate models like this? I am most familiar with ROC analysis and it seems there should be some analogous deterministic process using multiple metrics.

So far I've found Random Forests to be most similar as a modeling process but, from my understanding, each classifier is built on the same training set using different variables for decisions unlike my method. Would the validation methods still apply? Also, there is probably a name for what I am doing. Please forgive my lack of knowledge on this and feel free to inform me.

Further Reading

Currently, I'm applying an ad-hoc method but would like to have more insight for future applications:

Since the Classification model performed the best (according to ROC analysis)

• BAD_Posterior (from classification model) greater than B, flag as BAD
  • B was determined by finding the first point with a significant change in the TRUE Positive Rate
• GOOD_Posterior (from classification model) greater than G, flag as GOOD
  • G was determined by finding the first point with a significant change in the FALSE Negative rate

Further ROC analysis was run on all of the remaining unflagged records using both the GOOD and BAD Network models. Only one model (GOOD) was predictive so I optimized and applied additional rules based on the results.

All of this is good and well for what I am doing right now but it feels terribly unscientific and inefficient. There must be some methods I'm overlooking.

Answer 1

Given that what you are describing appears to be a binary classification problem, I don't see a point in having three different models for three different subsets of the data. The classical approach is to build a single model (e.g. logistic regression) trained on the whole data set (with cost-sensitive loss function in case of unbalanced classes) which typically produces decent results. More advanced methods such as generalized additive models that rely on bagging and/or boosting (e.g. Random Forests or Gradient Boosting Machine) may perform better if non-linearity and/or variable interactions are present.

Having multiple classifiers trained on subsets of data often helps in case of multi-class classification where a dedicated classifier is trained on a labeled subset of the data for a particular class (as in this paper)

Having said that, if you really need to combine multiple special-purpose classifiers, I would start here.