# Weighted Majority Rule

I have to implement a weighted majority vote which is an aggregation technique to generate one final prediction. In fact, several classifiers make local predictions. These are then collected and combined using a weighted majority rule to output the final prediction.

In this article, the soft voting is as follow:

$$\hat{y} = \arg \max_i \sum^{m}_{j=1} w_j p_{ij}$$

I didn't understand the predicted class probabilities for each classifier $p$. Does it mean the result of = occurence number of an activity / total number of instance in training set?

Is there a clear explanation for $p_{ij}$? I want to compute $p_{ij}$ manually.

• Thank you for your replies. My question is how to determine p{ij} value. In the example, 0.328 and 0.21 values, how was it computed ? In the first answer: "they use multiple prediction models to estimate the probabilities of these classes" => could you please propose an example of a model to estimate probabilities ? – Zoya Nov 7 '17 at 15:16
• @Zoya it this is unclear then I'm afraid that you need to check any handbook on machine learning, since it is the very basic issue. You should start with learning the basics before going any further. PS please do not use answers for comments. – Tim Nov 7 '17 at 15:20
• @Tim and I were writing our very similar answers simultaneously without knowing, and again, I would comment the same as Tim's comments above: any handbook on machine learning, but also those books on the basics of statistical models will be able to explain how any/most model(s) will be able to provide you with the probability of an 'outcome variable'. This means that without further context, any model which provides probability estimates will do. Moreover, the 'soft voting' from the article you refer to is specifically designed to combine the estimates of multiple models. – IWS Nov 8 '17 at 14:56

$p_{ij}$ is the probability assigned to $i$-th category by the $j$-th classifier. For example, you have a binary classification problem (cat vs. non-cat) and two classifiers: logistic regression and neural network with logistic link on the output layer. You make a prediction for some example and logistic regression says that the probability that it is a cat is 0.328, while neural network says that it is 0.21, not the weighted majority rule says that

\begin{align} \text{score}(\text{cat}) &= w_1 0.328 + w_2 0.21 \\ \text{score}(\text{non-cat}) &= w_1 (1-0.328) + w_2 (1-0.21) \end{align}

where $w_1$ and $w_2$ are weights applied to both classifiers. The class with greater score "wins" the competition and it taken as your classification.

For starters you need to realize their example applies to a situation where you'd want to predict the probability of a binary outcome. I.e. does the outcome occur (class '1'), or not (class '0')?

Additionally, they use multiple prediction models to estimate the probabilities of these classes.

Finally, using some weighting scheme to weigh these models' estimates, the weighted average probability of these estimates is used as the final estimated probability of class (0 or 1) membership.

So when referring to their formula:

• $y$ is the weighted average predicted probality of class membership
• $p_{ij}$ is the probability for individual $i$ for class membership of class '1', according to classifier/prediction model $j$.
• $w_j$ is the weight for prediction model $j$'s prediction
• for ${\operatorname{argmax}}$ see here

EDIT after clarification was asked how to estimate $p_{ij}$: any/most model(s) will be able to provide you with the probability of an 'outcome variable'. This means that without further context, any model which provides such estimates of probability will do (for example a logistic regression).

Finally, the 'soft voting' from the article you refer to is specifically designed to combine the estimates of multiple (different) models.