1
$\begingroup$

Hopefully this is now a much clearer picture of the question I want to answer:

For each of 60 different plans I ran 35 simulation replications of my Agent Based Model (ABM) simulation. Then for each simulation replication of each plan, I generated a score ($\geq0$), so that I ended up with 35 scores for each of the 60 plans. I now want to find the best plan which I define as the plan with the smallest mean score.

Of course I can compute the mean score for each one of the 60 plans, and I can rank each plan according to the mean score. At this point how do I test whether the 1st ranked mean score is statistically significantly different to the 2nd ranked score. And if I find it is not, how do I test whether the first two ranked mean scores are statistically significantly different to the 3rd ranked score, and so forth. Eventually I would like to end up with a set of plans (although the set can contain only one plan) for which I can state that the set of plans are equally best (as far as being statistically significantly the same).

This is what I was considering doing:

For each plan I compute the mean score and then select the 5 plans with the 5 smallest scores. I then use the Kruskal-Wallis H-Test to determine if any of the 5 means are significantly different (since my samples do not satisfy the normality or homoscedasticity assumptions of ANOVA).

If the p-value returned by the test is less than my significance level $\alpha$ then I know that at least one of the means is different and want to test for which one this is (are). At this point I need to use a post-hoc comparison test but am unsure as to which one I should use and why.

Improve this question
$\endgroup$
12
  • $\begingroup$ I'll just suggest searching for "ranking and selection" as the term that applies to this type of problem. $\endgroup$ – Russ Lenth Aug 26 '14 at 15:50
  • $\begingroup$ You seem to be throwing away the information needed to help solve your problem: by reducing each set of 35 replicates to a single score, you have eliminated all identifiable information about variation within the simulations. That leaves you powerless to attribute variations between the scores to anything but chance. As such, you cannot really conclude anything. Have you considered analyzing all $60\times 35$ values that you obtained, perhaps with an ANOVA, in order to assess the differences? $\endgroup$ – whuber♦ Aug 26 '14 at 15:54
  • $\begingroup$ PS I'll add that I don't see any reason at all to do the ANOVA (or K-W test). It's not necessary if your goal is just ranking the means (or medians?), and moreover it's pretty strange to do an ANOVA (or K-W) after using the data to select the treatments to include. $\endgroup$ – Russ Lenth Aug 26 '14 at 15:56
  • $\begingroup$ @whuber For each of the 60 plans I have 35 scores, one score per replication. I then want to see which plan has the (statistically significantly) smallest mean score. Does that make it clearer? $\endgroup$ – derNincompoop Aug 26 '14 at 15:59
  • 1
    $\begingroup$ @Russ You might want to expand a little on how you are interpreting this question. Your comments suggest you could be thinking of it differently than some other readers. For instance, in light of the OP's latest comment, would you object to, say, a Tukey HSD test? Because it is based on an ANOVA, it sounds like you would, but why? $\endgroup$ – whuber♦ Aug 26 '14 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.