0
$\begingroup$

Pardon my understanding of SVMs as it is very little.

We often hear of ensemble classifiers and stuff like this.

Say if i were to have 3 different SVM Models for the same dataset predicting a feature.

A test data point might have these results
Model A: Class 1
Model B: Class 2
Model C: Class 3

However, how would i know which class label should be the correct one given a test data point? It is often talked about that each Model will assign a score to its result and the Class label with the highest score would be the one chosen.

However, as far as i understand, SVM output would be a result of either 1 or -1. A binary SVM so as to say. Pardon my weak explanation of the question as i am not too sure how should i be asking it.

My Focus would be to ask how does an SVM model even assign a score to its prediction

$\endgroup$
2
$\begingroup$

The output of an SVM classifier is actually determined as $\hat y = \mathrm{sign}(\langle w, x\rangle + b)$, where $x$ is the test point and the model is defined by $w$ (the same kind of object as $x$) and $b\in \mathbb R$. The value $\langle w, x\rangle + b$ can be treated as a confidence score for the prediction; high absolute values correspond to confident predictions. This is sometimes called the "decision function", and can be useful to compare different examples with the same model. With multiple models, however, $w$ might vary substantially in norm (particularly if your models have different regularization parameters $C$), and so the magnitude of the scores can have very different meanings for different models; you can't necessarily compare them across models.

Platt scaling, as used by libsvm and other tools, converts this score (which can be any real number) to a probability (in $[0, 1]$) of being in the positive class. It does this by passing through a logistic function $\sigma(\langle w, x\rangle + b)$, with $\sigma(z) = 1 / (1 + \exp(A z + B))$. $A$ and $B$ are usually determined by cross-validation on the training set. Being probabilities, these are much more combinable/comparable/amenable to throwing into Bayesian models/whatever. These probabilities won't be completely reliable, but they're often at least sort of reasonable. Then you can use your favorite ensemble approach to make a final prediction.

$\endgroup$
1
$\begingroup$

There are a few parts to this answer.

First, you can have different types of SVM, like in SK-Learn, you have a Support Vector Regressor(SVR), and a Support Vector Classifier(SVC). The SVR can predict values on a number line, where the classifier will categorize results into classes. When you say "1 or -1" it sounds like you are using a classifier with only 2 classes, where you could be using many more.

Second, you have ensemble methods, there are many ways to combine the results, the score being one of them. There are also "majority rules" ensembles which pick the most commonly picked among all estimators and "distance based" ensembles which get a weighted value in-between all of the available predictions.

For a group of SVM, I would make a Majority Rules Ensemble with an odd number of SVM for a tie breaker. Then select the class picked as the majority.

Hope this helps.

$\endgroup$
  • $\begingroup$ Thats what i am unsure about. How does the 1 or -1 lead to a score. Is there something i am missing out. I am curious on how is the score even calculated $\endgroup$ – aceminer Mar 19 '15 at 7:22
  • 1
    $\begingroup$ @aceminer: AFAIK, e.g. libsvm uses a logistic regression to produce classification scores. $\endgroup$ – cbeleites Mar 19 '15 at 7:43
1
$\begingroup$

Ensemble classifiers typically use far larger numbers of models in the ensemble, say 100s to 1000s. You then tabulate the 100 or 1000 predictions against the classes as voting score. This is where e.g. the majority vote takes place.

Somewhat contrary to @ChisLucian's answer I'd put no particular emphasis on tie-breaking. In my experience it is more practical to have the classifier refusing to predict at all if the voting is not decided enough. Your 1 : 1 : 1 situation (as opposed to, say 2 : 1 : 0 or 0 : 3 : 0) basically means that the models are not sure: the votes are all over the place. So I'd answer with "not sure - no prediction possible".

If you happen to use a classifier that inherently predicts scores or even posterior probabilities (e.g. logistic regression), you can use different aggregation rules instead of the majority vote, e.g. use the median or average predicted score for each class and then go for the highest. The "not sure" option can be implemented there as well by requiring a threshold.


edited question: how does SVM produce a score output?

Does this paper answer your question: http://www.csie.ntu.edu.tw/~cjlin/papers/svmprob/svmprob.pdf

(no time right now to read and summarize - hopefully later)

IIRC e.g. libsv -b fits a logistic regression on top of the SVM (postprocessing?)

$\endgroup$
  • $\begingroup$ modified my question to ask it better. I think you are missing my point $\endgroup$ – aceminer Mar 19 '15 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.