My data involves testing 216 experimental units (computational models) for which one factor (the value of a specific shared parameter of the models) is administered in 4 treatments (i.e. the parameter possible values). The experimental design is repeated measures i.e. each model is tested with all the 4 values of the parameters. My observations corresponds to the "accuracy" of the model given a certain parameter value (or actually the AUC of the ROC analysis curve). There are no missing values, however there are many ties (i.e. many outcomes coincide).

My hypothesis is that the value of this parameter does not influence significantly the AUC of the model; or in other words, that the variation in the model's AUC given the parameter value is attributable to chance. In a sense, I will be expecting in a non-significant result of my hypothesis testing.


Now, following visual exploration of my data, and representation of the data distribution factorized by the treatment in the form of notched boxplots, intuitively any test would answer negatively (considerable overlap of the factorized distributions, especially on the notched region proportional to the standard error). However, when running Friedman in R I got the following results:

Friedman chi-squared = 50.3378, df = 3, p-value = 6.77e-11

Suspicious of this result, I tested in a tool I found online (http://vassarstats.net/index.html), and the results were:

Friedman chi-squared = 13.94, df = 3, p-value = 0.003

I found related information about this disagreement in here: http://r.789695.n4.nabble.com/Unexpected-behavior-in-friedman-test-and-ks-test-td902324.html

Still, based on the boxplots, I'm unconvinced that any test shall result in a significant finding. Perhaps the many ties are be fooling the implementations?. Also, I may be overpowering with too many observations, thus will obtain a significant result regardless of the data. The test assumptions seem to hold (unless I'm totally wrong). A bug in the R code seems unlikely after so many checkings (but can never be discarded).

My question is; in the light of the boxplots; why does Friedman gives a significant result? or am I doing something wrong? or am I being silly in applying Friedman when I should be applying other test? or should I trust this finding (blindly)?

I would appreciate plain and direct answers as I'm not an statistician.

  • $\begingroup$ One of the assumptions is of continuous distributions. I don't think that one holds. $\endgroup$ – Glen_b -Reinstate Monica Mar 7 '14 at 2:26
  • $\begingroup$ Please give the data itself. Not just a picture of them. $\endgroup$ – ttnphns Mar 7 '14 at 7:18
  • $\begingroup$ Thanks for your reply fellows! Glen_b; why do you feel that the continuous distribution assumption does not hold?. The independent variable (AUC) may range between 0 and 1 continuously (although in practice it ranges from 0.5 to 1). $\endgroup$ – user41471 Mar 8 '14 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.