How is a precision-recall curve calculated from an error matrix and vector of labels? The error matrix is of size NxN, and contains distances between the features associated with the respective row and column (the diagonal is consequently 0). The labels are in a 1xN vector, describing the class corresponding to the rows/columns of the error matrix, e.g. [1 1 1 1 2 2 2 ... 20 20 20 21 21 ... C-1 C-1 ... C C C], where C is the number of classes.
I've used the two equations below on all rank results for each class to end up with precision and recall values for each class.
Precision = TP / (TP + FP) Recall = TP / (TP + FN)
My question is (if what I'm doing is correct): is how are all the class precision-recall values combined/averaged to obtain a single plot? Do I just average all precision values at a given value of recall? Or is it acceptable to average all precision values for a given rank, and average all recall values for a given rank?
Thanks very much for any insight. The precision recall curves I get are shown in this image.
Code following (but probably unnecessary).
% returns the precision-recall points for plotting. % 'dismat' is an NxN distance matrix containing the distance between each feature and every other feature % 'labels' is an N-length vector that that corresponds to each row and column of 'dismat' function [prec,recall] = calcPrecisionRecall(dismat, labels) if size(labels, 1) ~= 1 labels = labels'; end numClasses = length(unique(labels)); [sortedDistMat, idx] = sort(dismat); % every column is sorted recalledClasses = labels(idx); recalledMatches = bsxfun(@eq, recalledClasses(2:end, :), labels); classMatches = zeros(size(recalledMatches, 1), numClasses); instancesPerClass = zeros(1, numClasses); for ii=1:numClasses classMatches(:, ii) = sum(recalledMatches(:, labels == ii), 2); instancesPerClass(ii) = sum(labels == ii); end % we'll just increase the rank to get the points prec = zeros(size(classMatches)); recall = zeros(size(classMatches)); sumClassMatches = cumsum(classMatches); rank = 1:size(classMatches, 1) for ci =1:numClasses prec(rank, ci) = sumClassMatches(rank, ci) ./ (rank' * instancesPerClass(ci)); recall(rank, ci) = sumClassMatches(rank, ci) / ((instancesPerClass(ci) - 1) * instancesPerClass(ci)); end % average all precision and all recall values (datapoints will therefore be avg prec--avg recall for a certain rank) avgPrec = mean(prec, 2); avgRecall = mean(recall, 2);
I previously asked this question on StackExchange (https://stackoverflow.com/questions/19824979/matlab-precision-recall-from-error-matrix) but was told to bring my question here.