I am trying to do a binary classification of text articles into {relevant, non-relevant}. The text articles have following features: [[article text, author & source, and image]]. Hence, I have built three classifiers each focussing on different features:

  1. first one is a NB Text classifier,
  2. second draws distributions over author & source, and
  3. third one is an image classifier.

Each of these classifiers return a "Probability of the article being relevant". If this probability is greater than 0.5 the article is classified as relevant, otherwise non-relevant.

Based on the classification success on validation data, I have identified the accuracy for each of these models.

Problem: For an incoming test article, each of these models generate some probability score of the article being relevant. I want to generate a final probability score taking into account the accuracy of each model.

What I Tried

  1. Currently, I am using a Normalized Weighted score using accuracy as weights.
  2. I also built up a NB model, conditioning on the output of each classifier, and assuming conditional independence for the outputs (which is a flawed assumption given the problem statement. That's why I am not much inclined towards using a NB model for this).

I feel I am just scratching the surface here and there must be good amount of literature on: 1) Merits/Demerits of working with multiple models, and 2) Combining output of multiple classifiers.

However, I am not able to reach the right set of articles (Not sure what to call such type of problems. Searching of ensemble/combination leads to Ensemble Learning (https://en.wikipedia.org/wiki/Ensemble_learning), which in my opinion is not exactly what I am trying to solve).

  • 4
    $\begingroup$ Here you can find, an article about different method that to do ensemble. mlwave.com/kaggle-ensembling-guide $\endgroup$
    – Romain
    Jan 17, 2016 at 18:22
  • $\begingroup$ @Romain: Can you please provide that link as an answer so that I can accept? Thank you. $\endgroup$ Mar 16, 2018 at 4:32

1 Answer 1


This may be helpful as well: Kuncheva, L. I. (2004). Combining pattern classifiers: methods and algorithms

Edit: For my similar problem, I ended up finding classifier probabilities based on accuracy values as described in the question here: Assigning probabilities to ensemble experts (classification) using using Theorem 4.2 (p. 127) from Kuncheva, L. I. (2004). Combining pattern classifiers: methods and algorithms, which says that the optimal combination weights for this case are $w_i=log(p_i/(1−p_i))$, where $p_i$ is the classification accuracy of the $i$-th expert. Then I converted weights to probabilities using projection onto probability simplex. Then one could combine those with the probabilities of each prediction that you already have to get the final probability values.

  • $\begingroup$ I think you should elaborate on your answer. $\endgroup$ May 15, 2019 at 18:51
  • $\begingroup$ sorry meant to post this as a comment. $\endgroup$ May 15, 2019 at 18:57
  • $\begingroup$ Just tried to do this but can't cause of low reputation. Perhaps I leave this answer here for now for future reference re literature, until someone copies puts it as a comment. $\endgroup$ May 15, 2019 at 19:07
  • $\begingroup$ Update - elaborated on the answer $\endgroup$ May 15, 2019 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.