I'm trying to figure out the best way to incorporate additional information into my predicted solution for a multi class classification problem. Here's the scenario.
There are 10 classes, 1-10. There is a training data set with features and the correct class label, and a testing data set for which I have the features but no class labels
The additional information I have is how many records in the test data have each class label. For example, I know Class 1 makes up 8% of the test data, Class 2 makes up 14% of the test data, etc. If it matters to the solution, I also know this distribution is different than the distribution of the training data (e.g Class 1 makes up 11% of the training data).
A perfect model built on the training data would correctly identify each record in the test data, and therefore predict that 8% of the test data is Class 1. I have a model (cross validated XGBoost, which is less than perfect) and a predicted probability that each record belongs to each class.
Now I have two pieces of information
- Predictions for each record in the test data
- The correct percent of the test data made up by each class.
How can I use the second piece of information to make my predictions more accurate?
I'd like to build it into the training process somehow or manipulate the predictions after the fact to reflect this information.