On combining SVMs Suppose we have a supervised training set $T=\{ (x_1, y_1),\dots, (x_n,y_n)\}$ where $x_i$ is an example and $y_i \in \{-1,+1\}$ is its label. Further suppose that examples are only observable through a feature  extraction function $f(x;s)$ where $x$ is an example and $s \in \{s_1,\dots,s_m\}$ is an argument for feature extraction. For each possible value of $s$, we train a linear SVM (on the set $\{ (f(x_1;s), y_1),\dots, (f(x_n;s),y_n)\}$). Let $w_i$ be learned weights of the SVM for $s=s_i$.
My question is on combining subsets of these SVMs for improved classification. Specifically, for a test example $x$, suppose that we have the scores of only the first two  SVMs (feature extraction is costly): $w_1^T f(x;s_1)$ and  $w_2^Tf(x;s_2)$. How can we combine these scores (optimally) to obtain a final decision? A trivial answer would be to train a SVM for each subset of $s$ values but this is not tractable. 
Ideally, I'm interested in a probabilistic interpretation. Assuming each SVM models $P(y|f(x;s_i))$, I want to express $P(y|f(x;s_1), f(x;s_2))$ using $P(y|f(x;s_1))$ and $P(y|f(x;s_2))$.
 A: You might find the following article helpful. Various techniques are outlined to get probability estimates for the outputs of SVM in Milgram.
In combining the probability estimates a weighted or unweighted sum of probabilities, naive Bayes or various other techniques can be used. See Chapter 5 for a comprehensive study of fusing classifier outputs. Kittler argues theoretically that the sum rule (adding up the probabilities of various classifiers and choosing the class with the highest probability) is optimal. 
I don't know what type of improvement in accuracy you can expect from only two support vector machines. The argument behind ensemble is that the probability of a correct collective decision approach 1 if the number of classifiers in the ensemble approach infinity. Using only two classifiers, will either agree on the decision or disagree on the decision. I would think that the ensemble wont be any better than the best single classifier?
A: Try 
A] Majority Voting 
B] Weighted Voting (Considering the distance to hyperplane as the weight or confidence of each hyperplane in their classification) 
C] AdaBoost [1] Algorithm.
[1] http://en.wikipedia.org/wiki/AdaBoost
