My data involves testing 216 experimental units (computational models) for which one factor (the value of a specific shared parameter of the models) is administered in 4 treatments (i.e. the parameter possible values). The experimental design is repeated measures i.e. each model is tested with all the 4 values of the parameters. My observations corresponds to the "accuracy" of the model given a certain parameter value (or actually the AUC of the ROC analysis curve). There are no missing values, however there are many ties (i.e. many outcomes coincide).
My hypothesis is that the value of this parameter does not influence significantly the AUC of the model; or in other words, that the variation in the model's AUC given the parameter value is attributable to chance. In a sense, I will be expecting in a non-significant result of my hypothesis testing.
Now, following visual exploration of my data, and representation of the data distribution factorized by the treatment in the form of notched boxplots, intuitively any test would answer negatively (considerable overlap of the factorized distributions, especially on the notched region proportional to the standard error). However, when running Friedman in R I got the following results:
Friedman chi-squared = 50.3378, df = 3, p-value = 6.77e-11
Suspicious of this result, I tested in a tool I found online (http://vassarstats.net/index.html), and the results were:
Friedman chi-squared = 13.94, df = 3, p-value = 0.003
I found related information about this disagreement in here: http://r.789695.n4.nabble.com/Unexpected-behavior-in-friedman-test-and-ks-test-td902324.html
Still, based on the boxplots, I'm unconvinced that any test shall result in a significant finding. Perhaps the many ties are be fooling the implementations?. Also, I may be overpowering with too many observations, thus will obtain a significant result regardless of the data. The test assumptions seem to hold (unless I'm totally wrong). A bug in the R code seems unlikely after so many checkings (but can never be discarded).
My question is; in the light of the boxplots; why does Friedman gives a significant result? or am I doing something wrong? or am I being silly in applying Friedman when I should be applying other test? or should I trust this finding (blindly)?
I would appreciate plain and direct answers as I'm not an statistician.
