# What do "real values" refer to in supervised classification?

I'm using supervised classification algorithms from mlpy to classify things into two groups for a question-answering system. I don't really know how these algorithms work, but they seem to be doing vaguely what I want.

I would like to get some measure of confidence out of the classifiers. I can get "real-valued predictions" from the classifiers. These appear to be values of what I would call a link function. Here's some sample output from my system.

Predictions as to whether an answer is correct
for random data from various models

Results for one run
----------------------------------------------
Model Result Confidence? ("Real value")
----- ------ ---------------------------------
SVM  [True, 0.10396502611075412]
FDA  [True, 3.3052963597375227]
SRDA  [False, 0.34205901959526142]
PDA  [True, 3.8857018468328794]
----------------------------------------------

Results for another run
----------------------------------------------
Model Result Confidence? ("Real value")
----- ------ ---------------------------------
SVM  [False, -0.0059697528841203369]
FDA  [False, -0.15660355802446979]
SRDA  [False, 1.2465697042600801]
PDA  [True, 0.23122963338708608]
----------------------------------------------


The real values are generally more positive for classifications as "True" and more negative for classifications as "False", but the link is a bit more complex than that. Can I turn these real values into confidence measures? If so, how?

And I'm playing with the following classifiers.

• Support Vector Machines (SVMs)
• K Nearest Neighbor (KNN)
• Fisher Discriminant Analysis (FDA)
• Spectral Regression Discriminant Analysis (SRDA)
• Penalized Discriminant Analysis (PDA)
• Diagonal Linear Discriminant Analysis (DLDA)

Update: Having thought about this more, realize that I just need the rank of the confidence actually. Does it seem right the answers with the highest real values be the ones that are most confidently categorized in the True group? I'd still like to understand this a bit better, but an answer to that would be nice in the short term.

• I suggest to change "real values" in the question header to "scores" or "confidences", since "real values" is a rather arbitrary term (as explained in my response). May 6 '11 at 11:21
• I think I should keep "real values" because I might not have had the question had the documentation used less arbitrary terms. May 6 '11 at 12:57
• good point May 6 '11 at 14:31

Regarding the "Real Values"

The "Real Values" are better called "confidences" or (from my pov the most common term) "scores".

Such scores are often normalized so that they sum up to 1 for all classes. They represent a measure how, well, confident the model is that the presented example belongs to a certain class. They are highly dependent on the general strategy and the properties of the algorithm. For example in KNN the score for a class i is calculated by averaging the distance to those examples which both belong to the k-nearest-neighbors and have class i. Then the score is sum-normalized across all classes.