I have a large dataset consisting of many features for each record which includes either a GOOD or BAD outcome. Three metrics have been created for classification purposes using three respective Bayesian Belief Networks:
- Classification Network including outcome variable trained on both GOOD and BAD records
- GOOD Network trained on only GOOD records
- BAD Network trained on only BAD records
For the GOOD and BAD Networks, a most likely state is predicted for each feature given information on all other features and compared with the actual value present. The GOOD and BAD metrics are then defined by the percent of features correctly predicted.
Are there any standard methods that exist for defining optimal thresholds and rule sets for classification on separate models like this? I am most familiar with ROC analysis and it seems there should be some analogous deterministic process using multiple metrics.
So far I've found Random Forests to be most similar as a modeling process but, from my understanding, each classifier is built on the same training set using different variables for decisions unlike my method. Would the validation methods still apply? Also, there is probably a name for what I am doing. Please forgive my lack of knowledge on this and feel free to inform me.
Currently, I'm applying an ad-hoc method but would like to have more insight for future applications:
Since the Classification model performed the best (according to ROC analysis)
- BAD_Posterior (from classification model) greater than B, flag as BAD
- B was determined by finding the first point with a significant change in the TRUE Positive Rate
- GOOD_Posterior (from classification model) greater than G, flag as GOOD
- G was determined by finding the first point with a significant change in the FALSE Negative rate
Further ROC analysis was run on all of the remaining unflagged records using both the GOOD and BAD Network models. Only one model (GOOD) was predictive so I optimized and applied additional rules based on the results.
All of this is good and well for what I am doing right now but it feels terribly unscientific and inefficient. There must be some methods I'm overlooking.