I'm trying to figure out the best way to incorporate additional information into my predicted solution for a multi class classification problem. Here's the scenario.

There are 10 classes, 1-10. There is a training data set with features and the correct class label, and a testing data set for which I have the features but no class labels

The additional information I have is how many records in the test data have each class label. For example, I know Class 1 makes up 8% of the test data, Class 2 makes up 14% of the test data, etc. If it matters to the solution, I also know this distribution is different than the distribution of the training data (e.g Class 1 makes up 11% of the training data).

A perfect model built on the training data would correctly identify each record in the test data, and therefore predict that 8% of the test data is Class 1. I have a model (cross validated XGBoost, which is less than perfect) and a predicted probability that each record belongs to each class.

Now I have two pieces of information

  1. Predictions for each record in the test data
  2. The correct percent of the test data made up by each class.

How can I use the second piece of information to make my predictions more accurate?

I'd like to build it into the training process somehow or manipulate the predictions after the fact to reflect this information.

  • 1
    $\begingroup$ You should be able to do this with Bayes' Theorem. There is a standard solution to this with, for example, Linear Discriminant Analysis and MNL. However, I have no idea what XGBoost is, so am not game to post a more precise answer. $\endgroup$ – Tim Mar 7 '17 at 5:40
  • $\begingroup$ @Tim would the procedure you mention be applied after the original predictions are made, or as part of a model building process? I am not too familiar with LDA or MNL, so if you have resources to help me understand how to apply them to the problem at hand that would be great. $\endgroup$ – Caleb Mar 7 '17 at 8:12
  • $\begingroup$ The basic mechanics of both LDA and MNL are that once you have fit the model, you can change a specific set of parameters that address the prior probability of class membership in the predictions. Let me spend a few minutes of contemplation and see if I can figure out the general solution. $\endgroup$ – Tim Mar 7 '17 at 21:51
  • 1
    $\begingroup$ See equation (4) in pdfs.semanticscholar.org/d6d2/… $\endgroup$ – Tim Mar 7 '17 at 22:00
  • $\begingroup$ @Tim, thanks for the link to the paper.Implementing this helped me very minimally, but there was a tiny bit of lift. It's basically exactly what I need, except, when I adjust my predictions I still have total prediction for class x = 500 observations and I know class x has 600 observations in the test set. I suppose any further manipulation beyond the Bayes formula given would result in a loss of classification accuracy. Perhaps iterating through until reaching a steady state would be a good solution. $\endgroup$ – Caleb Mar 8 '17 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.