1
$\begingroup$

How is a precision-recall curve calculated from an error matrix and vector of labels? The error matrix is of size NxN, and contains distances between the features associated with the respective row and column (the diagonal is consequently 0). The labels are in a 1xN vector, describing the class corresponding to the rows/columns of the error matrix, e.g. [1 1 1 1 2 2 2 ... 20 20 20 21 21 ... C-1 C-1 ... C C C], where C is the number of classes.

I've used the two equations below on all rank results for each class to end up with precision and recall values for each class.

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

My question is (if what I'm doing is correct): is how are all the class precision-recall values combined/averaged to obtain a single plot? Do I just average all precision values at a given value of recall? Or is it acceptable to average all precision values for a given rank, and average all recall values for a given rank?

Thanks very much for any insight. The precision recall curves I get are shown in this image. enter image description here

Code following (but probably unnecessary).

% returns the precision-recall points for plotting. 
% 'dismat' is an NxN distance matrix containing the distance between each feature and every other feature
% 'labels' is an N-length vector that that corresponds to each row and column of 'dismat'
function [prec,recall] = calcPrecisionRecall(dismat, labels)

if size(labels, 1) ~= 1
    labels = labels';
end

numClasses = length(unique(labels));

[sortedDistMat, idx] = sort(dismat);   % every column is sorted
recalledClasses = labels(idx);

recalledMatches = bsxfun(@eq, recalledClasses(2:end, :), labels);

classMatches = zeros(size(recalledMatches, 1), numClasses);
instancesPerClass = zeros(1, numClasses);
for ii=1:numClasses
    classMatches(:, ii) = sum(recalledMatches(:, labels == ii), 2);
    instancesPerClass(ii) = sum(labels == ii);
end

% we'll just increase the rank to get the points
prec = zeros(size(classMatches));
recall = zeros(size(classMatches));
sumClassMatches = cumsum(classMatches);
rank = 1:size(classMatches, 1)
for ci =1:numClasses
    prec(rank, ci) = sumClassMatches(rank, ci) ./ (rank' * instancesPerClass(ci));
    recall(rank, ci) = sumClassMatches(rank, ci) / ((instancesPerClass(ci) - 1) * instancesPerClass(ci));
end

% average all precision and all recall values (datapoints will therefore be avg prec--avg recall for a certain rank)
avgPrec = mean(prec, 2);
avgRecall = mean(recall, 2);

I previously asked this question on StackExchange (https://stackoverflow.com/questions/19824979/matlab-precision-recall-from-error-matrix) but was told to bring my question here.

$\endgroup$
  • $\begingroup$ To compute a precision-recall curve, you need to have for each data point of your training set the probability of belonging in the positive class. Where are these probabilities stored? $\endgroup$ – Franck Dernoncourt Nov 9 '13 at 17:04
  • $\begingroup$ I don't have the probabilities stored anywhere. I have the NxN error/dissimilarity matrix, and a vector of ground-truth labels. I've used the ground-truths to provide values for TP, FN, FP for each class at each rank. $\endgroup$ – James Brown Nov 9 '13 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.