2
$\begingroup$

I have trained an linear SVM which takes a pair of objects, computes features and is expected to learn a semantic similarity function between objects(we can say that it predicts whether the two objects are similar enough that they should be merged or not). The problem I am facing is that the predictions can be from $-\infty$ to $\infty$ and I need a score from [0,1] as a semantic similarity.

One suggestion that I received was that use min-max normalization to normalize the scores. Is there a better way(which is more generic rather than depending on values of min and max from training data)? Please mention the assumptions also, in case your method has any.

Thanks

$\endgroup$
  • 2
    $\begingroup$ Are these predictions on new data not in the training set? If so, have a look at this paper. $\endgroup$ – learner Apr 3 '13 at 17:20
  • $\begingroup$ When you say you're learning a similarity function, how are you doing that in the SVM framework? A typical classification SVM will give you a real-numbered decision value, whose sign is a class label prediction, for a single object. It's not a similarity function mapping a pair of objects to a similarity score, unless your training/test inputs are actually pairs of objects labeled by their similarity. $\endgroup$ – djs Apr 3 '13 at 22:50
4
$\begingroup$

Use a sigmoid function $y = \frac{1}{(1+e^{-f(x)})}$ where $f(x)$ is your output [-inf, inf], then y will be between [0,1]. This is a known techinique, see for example multilayer perceptron (MLP)

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

The paper suggested by learner above is a good suggestion of how this can be accomplished. I would also suggest using the leave-one-out cross-validation predictions to fit the sigmoid function, rather than the actual output on the training examples as extra insurance against over-fitting the sigmoid.

However, if you are interested in more than binary classification, then you might be better off using kernel logistic regression instead of the SVM, where the output is an estimate of a-posteriori probability of class membership, and so will be in the range [0-1]. However the training criterion used for KLR takes the who range of probability into account rather than concentrating on the decision boundary.

Prof. Vapnik quite rightly says that if you want to solve a problem, you ought to choose a method that solves it directly, rather than solving a more general problem and then simplifying the solution (as the former approach concentrates on what is actually important). I would argue that you should also avoid solving a more simple problem and then post-processing it to approximate the solution to a more general problem - which is essentially what we are doing when fitting a sigmoid to the output of an SVM. If we want probabilities, we should estimate them directly using KLR; if we want binary classification, we should use and SVM, rather than thresholding the probabilities provided by KLR.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Thanks to @learner, I am using the paper by Lin. The paper approximates the svm posterior using a sigmoid model.

$$Pr(y=+1~|~x) \sim \frac{1}{1+exp[Af(x)+B]} $$

Parameters A and B are learnt from a labelled dataset. To avoid overfitting an out-of-sample model is used. Also, to avoid bias, instead of using the training dataset, an independent dataset is used for calibration. For details, refer my thesis.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The link to your thesis 404s. $\endgroup$ – dmh Sep 22 '15 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.