5
$\begingroup$

I uploaded the data and then I used bootstrapping to have 10 different samples from the original data but with the same length as the original data. For each sample, I used 7 distance metrics, and I calculated accuracy and other performance measures.

First, I am trying to compare 7 different accuracies using Friedman test. 

Accuracies Matrix
           [,1]      [,2]      [,3]      [,4]      [,5]      [,6]      [,7]
 [1,] 0.9753954 0.9771529 0.9789104 0.9789104 0.9806678 0.9771529 0.9806678
 [2,] 0.9736380 0.9806678 0.9806678 0.9806678 0.9841828 0.9771529 0.9771529
 [3,] 0.9753954 0.9841828 0.9806678 0.9771529 0.9806678 0.9771529 0.9718805
 [4,] 0.9771529 0.9859402 0.9789104 0.9789104 0.9824253 0.9824253 0.9841828
 [5,] 0.9736380 0.9806678 0.9771529 0.9824253 0.9824253 0.9806678 0.9771529
 [6,] 0.9701230 0.9789104 0.9736380 0.9806678 0.9841828 0.9824253 0.9753954
 [7,] 0.9912127 0.9912127 0.9859402 0.9859402 0.9859402 0.9841828 0.9824253
 [8,] 0.9789104 0.9806678 0.9859402 0.9859402 0.9841828 0.9806678 0.9789104
 [9,] 0.9806678 0.9841828 0.9876977 0.9824253 0.9841828 0.9859402 0.9841828
[10,] 0.9789104 0.9771529 0.9753954 0.9789104 0.9666081 0.9613357 0.9630931

I got the following result: 

    Friedman rank sum test

data:  Datam
Friedman chi-squared = 16.252, df = 6, p-value = 0.01246

That means there is a significant difference between the accuracy groups. So I used the function posthoc.friedman.nemenyi.test from R's PMCMR package to determine which pairs are significantly different and I got the following:

    Pairwise comparisons using Nemenyi multiple comparison test 
             with q approximation for unreplicated blocked data 

data:  Accuracies Matrix 

     [,1]  [,2]  [,3]  [,4]  [,5]  [,6] 
[1,] 0.088 -     -     -     -     -    
[2,] 0.310 0.998 -     -     -     -    
[3,] 0.185 1.000 1.000 -     -     -    
[4,] 0.027 1.000 0.958 0.991 -     -    
[5,] 0.804 0.830 0.987 0.946 0.576 -    
[6,] 0.987 0.436 0.804 0.645 0.207 0.996

P value adjustment method: none

How to interpret the result of posthoc.friedman.nemenyi.test?

Improve this question
$\endgroup$
22
  • 1
    $\begingroup$ What are the actual data? How is the Accuracies Matrix set up? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 26, 2018 at 19:53
  • $\begingroup$ @gung♦, I calculated the accuracies from different model then I put them in a matrix. $\endgroup$
    – jeza
    Commented Sep 26, 2018 at 20:03
  • 1
    $\begingroup$ What are they? Are they outputs from some models? For what? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 26, 2018 at 20:26
  • $\begingroup$ @gung♦, yes it is accuracies outputs from different models. $\endgroup$
    – jeza
    Commented Sep 26, 2018 at 20:29
  • 1
    $\begingroup$ What are the models? What are the models' accuracies? Are they classifying objects that they get right or wrong? Something else? What is your situation? What are your data? What are you trying to do? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 26, 2018 at 20:32

1 Answer 1

Reset to default
7
+50
$\begingroup$

The problem is that the row and column labels for the matrix make the results difficult to understand. ‡

In the following, since there are no column labels, the columns will be labeled V1 to V7 by default. This will make it easy to evaluate the comparisons between them.

if(!require(PMCMR)){install.packages("PMCMR")}

Input =("
0.9753954 0.9771529 0.9789104 0.9789104 0.9806678 0.9771529 0.9806678
0.9736380 0.9806678 0.9806678 0.9806678 0.9841828 0.9771529 0.9771529
0.9753954 0.9841828 0.9806678 0.9771529 0.9806678 0.9771529 0.9718805
0.9771529 0.9859402 0.9789104 0.9789104 0.9824253 0.9824253 0.9841828
0.9736380 0.9806678 0.9771529 0.9824253 0.9824253 0.9806678 0.9771529
0.9701230 0.9789104 0.9736380 0.9806678 0.9841828 0.9824253 0.9753954
0.9912127 0.9912127 0.9859402 0.9859402 0.9859402 0.9841828 0.9824253
0.9789104 0.9806678 0.9859402 0.9859402 0.9841828 0.9806678 0.9789104
0.9806678 0.9841828 0.9876977 0.9824253 0.9841828 0.9859402 0.9841828
0.9789104 0.9771529 0.9753954 0.9789104 0.9666081 0.9613357 0.9630931
")
Matrix = as.matrix(read.table(textConnection(Input)))

Matrix

   ###             V1        V2        V3        V4        V5        V6        V7
   ### [1,] 0.9753954 0.9771529 0.9789104 0.9789104 0.9806678 0.9771529 0.9806678
   ### [2,] 0.9736380 0.9806678 0.9806678 0.9806678 0.9841828 0.9771529 0.9771529
   ### [3,] 0.9753954 0.9841828 0.9806678 0.9771529 0.9806678 0.9771529 0.9718805
   ### [4,] 0.9771529 0.9859402 0.9789104 0.9789104 0.9824253 0.9824253 0.9841828
   ### [5,] 0.9736380 0.9806678 0.9771529 0.9824253 0.9824253 0.9806678 0.9771529
   ### [6,] 0.9701230 0.9789104 0.9736380 0.9806678 0.9841828 0.9824253 0.9753954
   ### [7,] 0.9912127 0.9912127 0.9859402 0.9859402 0.9859402 0.9841828 0.9824253
   ### [8,] 0.9789104 0.9806678 0.9859402 0.9859402 0.9841828 0.9806678 0.9789104
   ### [9,] 0.9806678 0.9841828 0.9876977 0.9824253 0.9841828 0.9859402 0.9841828
   ### [10,] 0.9789104 0.9771529 0.9753954 0.9789104 0.9666081 0.9613357 0.9630931

library(PMCMR)

posthoc.friedman.nemenyi.test(Matrix)

   ###  Pairwise comparisons using Nemenyi multiple comparison test 
             with q approximation for unreplicated blocked data 

   ### data:  Matrix 

   ###    V1    V2    V3    V4    V5    V6   
   ### V2 0.088 -     -     -     -     -    
   ### V3 0.310 0.998 -     -     -     -    
   ### V4 0.185 1.000 1.000 -     -     -    
   ### V5 0.027 1.000 0.958 0.991 -     -    
   ### V6 0.804 0.830 0.987 0.946 0.576 -    
   ### V7 0.987 0.436 0.804 0.645 0.207 0.996
   ###
   ### P value adjustment method: none

The output above is a table of p-values, each comparing two groups. If you are using p = 0.05 as your cutoff, the only significant comparison is V1 vs. V5 (p = 0.027). The rest of the p-values are all greater than 0.05.

It may be useful to translate this matrix of p-values to a compact letter display. In this output, groups sharing a letter are not significantly different. For this I'll use the fullPTable function in the rcompanion package † and multcompLetters from multcompView.

if(!require(multcompView)){install.packages("multcompView")}
if(!require(PMCMR)){install.packages("PMCMR")}
if(!require(rcompanion)){install.packages("rcompanion")}

library(PMCMR)
library(rcompanion)
library(multcompView)

PT  = posthoc.friedman.nemenyi.test(Matrix)$p.value
PT1 = fullPTable(PT)
PT1
library(multcompView)
multcompLetters(PT1)

   ###    V1   V2   V3   V4   V5   V6   V7 
   ###   "a" "ab" "ab" "ab"  "b" "ab" "ab"

V1 and V5 are the only two groups not sharing a letter.

Addition: PMCMRplus package

There are a few different post-hoc tests available for Friedman's test in PMCMRplus package. Functions begin with frdAllPairs. The Nemenyi test appears to produce results similar to those above. For this example, it was necessary to add row labels to the matrix.

if(!require(PMCMRplus)){install.packages("PMCMRplus")}

library(PMCMRplus)

rownames(Matrix) = LETTERS[1:10]

frdAllPairsNemenyiTest(Matrix)

   # Pairwise comparisons using Nemenyi-Wilcoxon-Wilcox all-pairs test for a two-way balanced complete block design
   # 
   #    V1    V2    V3    V4    V5    V6   
   # V2 0.088 -     -     -     -     -    
   # V3 0.310 0.998 -     -     -     -    
   # V4 0.185 1.000 1.000 -     -     -    
   # V5 0.027 1.000 0.958 0.991 -     -    
   # V6 0.804 0.830 0.987 0.946 0.576 -    
   # V7 0.987 0.436 0.804 0.645 0.207 0.996
   # 
   # P value adjustment method: single-step

‡ Note: This answer addresses the primary question: conducting and interpreting Nemenyi test. It does not weigh in on the discussion in the comments, as to whether the generation of this data makes sense or if Friedman's test is the applicable test in this case.

† Caveat: I am the author of this package.

Improve this answer
$\endgroup$
17
  • 2
    $\begingroup$ To make things simple, yes, V5 is greater than V1. But that's not really the correct interpretation of Friedman test and post-hocs. A better interpretation is something like, within each row, V5 tends to be greater than V1. This considers the ranks of the data and not the values themselves. $\endgroup$
    – Sal Mangiafico
    Commented Sep 30, 2018 at 0:08
  • 2
    $\begingroup$ Two reasons: 1) Friedman test doesn't look at the actual values, but the values ranked relative to each other; 2) Friedman test evaluates the differences between groups within each block. So... saying V1 is greater than V1 is one way to report the results simply, but it's not really what the test is testing for. $\endgroup$
    – Sal Mangiafico
    Commented Sep 30, 2018 at 14:10
  • 2
    $\begingroup$ Friedman test doesn't really compare medians. If you wanted to compare medians, you would use a different test. That being said, it makes sense to report medians with Friedman, at least in most usual circumstances. The medians make sense with your data: apply(Matrix, 2, FUN = median) and plot(apply(Matrix, 2, FUN = median)) $\endgroup$
    – Sal Mangiafico
    Commented Sep 30, 2018 at 15:12
  • 1
    $\begingroup$ I don't follow the question. You conducted the Nemenyi to compare the groups, didn't you? If you want to compare medians per se, you would probably want to use a different test. $\endgroup$
    – Sal Mangiafico
    Commented Sep 30, 2018 at 15:28
  • 2
    $\begingroup$ Reporting the medians is useful for your audience to get some sense of the differences among the groups. Likewise, showing a histogram for each, or a box plot for each is useful. None of these captures the way Friedman test is treating the data as blocked, or the way it is handling ranks, so in extreme cases these methods could be misleading. But they're usually pretty helpful for the audience. $\endgroup$
    – Sal Mangiafico
    Commented Sep 30, 2018 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.