# On combining SVMs

Suppose we have a supervised training set $T=\{ (x_1, y_1),\dots, (x_n,y_n)\}$ where $x_i$ is an example and $y_i \in \{-1,+1\}$ is its label. Further suppose that examples are only observable through a feature extraction function $f(x;s)$ where $x$ is an example and $s \in \{s_1,\dots,s_m\}$ is an argument for feature extraction. For each possible value of $s$, we train a linear SVM (on the set $\{ (f(x_1;s), y_1),\dots, (f(x_n;s),y_n)\}$). Let $w_i$ be learned weights of the SVM for $s=s_i$.

My question is on combining subsets of these SVMs for improved classification. Specifically, for a test example $x$, suppose that we have the scores of only the first two SVMs (feature extraction is costly): $w_1^T f(x;s_1)$ and $w_2^Tf(x;s_2)$. How can we combine these scores (optimally) to obtain a final decision? A trivial answer would be to train a SVM for each subset of $s$ values but this is not tractable.

Ideally, I'm interested in a probabilistic interpretation. Assuming each SVM models $P(y|f(x;s_i))$, I want to express $P(y|f(x;s_1), f(x;s_2))$ using $P(y|f(x;s_1))$ and $P(y|f(x;s_2))$.

## 2 Answers

You might find the following article helpful. Various techniques are outlined to get probability estimates for the outputs of SVM in Milgram.

In combining the probability estimates a weighted or unweighted sum of probabilities, naive Bayes or various other techniques can be used. See Chapter 5 for a comprehensive study of fusing classifier outputs. Kittler argues theoretically that the sum rule (adding up the probabilities of various classifiers and choosing the class with the highest probability) is optimal.

I don't know what type of improvement in accuracy you can expect from only two support vector machines. The argument behind ensemble is that the probability of a correct collective decision approach 1 if the number of classifiers in the ensemble approach infinity. Using only two classifiers, will either agree on the decision or disagree on the decision. I would think that the ensemble wont be any better than the best single classifier?

• As an archival note: Section 5.4.2 of Kuncheva's book gives a Bayesian validation for the sum rule proposed by Kittler et al. – emrea Mar 8 '13 at 19:16

Try
A] Majority Voting
B] Weighted Voting (Considering the distance to hyperplane as the weight or confidence of each hyperplane in their classification)
C] AdaBoost  Algorithm.