4
$\begingroup$

I am comparing the performance of five classifiers over 100 datasets. I have a table of data where each column is a classifier and each row is a dataset, so the cell (i,j) is the performance (accuracy) of classifier j over sample i.

I am using the Friedman test for statistical comparison, but my question relates to visualization.

I want to generate some sort of plot that gives an indication of which classifier performs best.

So far, my idea has been to separate the original matrix in five 100x2 matrixes, each corresponding to a classifier, where the first column indicates the dataset's label (1-100) and the second column the performance.

I then sort all columns in ascending order based on the values in the second column, so for example:

1 80
2 85
3 79

Becomes

3 79
2 85
1 80

I then plot the sorted performance vector for each classifier, such that in the resulting plot I can compare best performances, second-best performances, etc.

This allows me to draw conclusions such as "The best performance of Classifier A is better than the best performance of Classifier B."

However, I am not convinced this provides meaningful information. For example, if the best performance of A was on dataset 1 and the best performance of B on dataset 2, it might still be that B outpeformed A on dataset 2.

Any advice?

$\endgroup$
2
$\begingroup$

You could use a heatmap (matrix) where each column is a classifier and each row a dataset. For each row you rank your classifiers best to worst and assign a color gradient (green to red for instance).

Then ideally you do not display classifiers (columns) randomly but from the best one (highest average or median score if they are comparable between data set) to the worst one... You should see a matrix with green on the left and red on the right if few classifiers are constantly doing better than others. if you see some green in the middle of red it means that, as you said in your post, on this specific dataset a given classifier is doing very well.

$\endgroup$
  • $\begingroup$ Thanks @Romain, I tried that and it seems to work really well, I'll present that together with a box plot and I think it should work out well :) $\endgroup$ – MrD Dec 14 '15 at 15:50
1
$\begingroup$

Another option is to create a plot with x axis being the datasets (preferably sorted by "difficulty") and classifier performance on the y axis. Then draw 5 differently-colored lines for each classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.