Suppose we have a supervised training set $T=\{ (x_1, y_1),\dots, (x_n,y_n)\}$ where $x_i$ is an example and $y_i \in \{-1,+1\}$ is its label. Further suppose that examples are only observable through a feature extraction function $f(x;s)$ where $x$ is an example and $s \in \{s_1,\dots,s_m\}$ is an argument for feature extraction. For each possible value of $s$, we train a linear SVM (on the set $\{ (f(x_1;s), y_1),\dots, (f(x_n;s),y_n)\}$). Let $w_i$ be learned weights of the SVM for $s=s_i$.

My question is on combining subsets of these SVMs for improved classification. Specifically, for a test example $x$, suppose that we have the scores of only the first two SVMs (feature extraction is costly): $w_1^T f(x;s_1)$ and $w_2^Tf(x;s_2)$. How can we combine these scores (optimally) to obtain a final decision? A trivial answer would be to train a SVM for each subset of $s$ values but this is not tractable.

Ideally, I'm interested in a probabilistic interpretation. Assuming each SVM models $P(y|f(x;s_i))$, I want to express $P(y|f(x;s_1), f(x;s_2))$ using $P(y|f(x;s_1))$ and $P(y|f(x;s_2))$.


2 Answers 2


You might find the following article helpful. Various techniques are outlined to get probability estimates for the outputs of SVM in Milgram.

In combining the probability estimates a weighted or unweighted sum of probabilities, naive Bayes or various other techniques can be used. See Chapter 5 for a comprehensive study of fusing classifier outputs. Kittler argues theoretically that the sum rule (adding up the probabilities of various classifiers and choosing the class with the highest probability) is optimal.

I don't know what type of improvement in accuracy you can expect from only two support vector machines. The argument behind ensemble is that the probability of a correct collective decision approach 1 if the number of classifiers in the ensemble approach infinity. Using only two classifiers, will either agree on the decision or disagree on the decision. I would think that the ensemble wont be any better than the best single classifier?

  • 1
    $\begingroup$ As an archival note: Section 5.4.2 of Kuncheva's book gives a Bayesian validation for the sum rule proposed by Kittler et al. $\endgroup$
    – emrea
    Mar 8, 2013 at 19:16

A] Majority Voting
B] Weighted Voting (Considering the distance to hyperplane as the weight or confidence of each hyperplane in their classification)
C] AdaBoost [1] Algorithm.

[1] http://en.wikipedia.org/wiki/AdaBoost


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.