4
$\begingroup$

We have recently conducted a cluster analysis for an open card sorting task in which 19 experts grouped 112 items each. We achieved decent silhouette values for the cluster analysis but have been asked by reviewers to provide evidence that the number of experts we used was sufficient for this task (I think they are expecting something kind of power analysis).

We have used Kendall's Coefficient of Concordance to examine the concordance in individual sort patterns between participants (this came out as .25), but it seems they want something more. After looking around I cannot find anything which might suggest a suitable sample size or even that sample size is not important and I'm pretty sure power analyses don't apply for cluster analysis. Is there anything I can report to show my sample size is sufficient (or not if the case may be)? I am using R to analyse the data.

BTW by sample size I mean number of participants not number of items included in the cluster analysis.

Improve this question
$\endgroup$
5
  • $\begingroup$ I could not see cluster analysis in what you've done. Is that grouping of cards by experts you call cluster analysis? Or, if you did cluster analysis on the results of that grouping please provide more details about that analysis. $\endgroup$
    – ttnphns
    Commented Nov 26, 2011 at 8:04
  • $\begingroup$ @ttnphns Sorry, I should have added more detail. The participants are asked to group 112 items. We created a matrix for each participant from these groupings assigning 0 pairs to items which were assigned to the same group and 1 to pairs of items in different groups. So we had a binary matrix (a simple distance matrix) for all possible item pairings for each participant. These were then averaged across participants to created one aggregated distance matrix. This aggregated distance matrix was then submitted for cluster analysis (fuzzy CA) to identify the grouping of items across participants. $\endgroup$
    – Jimichanga1
    Commented Nov 26, 2011 at 21:34
  • $\begingroup$ I see. Thanks. Cluster analysis is not inferential technique, so question of power cannot be arised. However, if your experts are in a good agreement according to Kendall W that you used (and there were no unusual experts-outliers), than the averaging of the matrices into one and doing clustering of it is warranted whatever the number of experts is. The question of the number of experts (the sample size) is important, as always, but is unrelated to the cluster analysis. $\endgroup$
    – ttnphns
    Commented Nov 27, 2011 at 7:12
  • $\begingroup$ If I understand correctly, each participant created a partition of the data in groups or clusters. You can look at mutual consistency between participants by considering distances between clusterings (I prefer a metric distance like 'variance of information' over indexes such as Rand). This would give you a participant/participant (dis)agreement matrix. For a unified clustering there is a lot published about 'consensus clustering'. Unfortunately this does not answer your question, but relates to a stage of your research preceding that question. $\endgroup$
    – micans
    Commented Nov 28, 2011 at 13:56
  • $\begingroup$ Thanks so much for your answers. I think I have enough information to formulate a response to the reviewers now. I will definitely be looking up 'consensus clustering' seems very interesting. $\endgroup$
    – Jimichanga1
    Commented Dec 8, 2011 at 16:10

1 Answer 1

Reset to default
0
$\begingroup$

It sounds like they want you to do a post hoc power analysis. The purpose of a power analysis is to optimize the study that will be conducted (e.g., to increase the likelihood of getting a statistically significant result). A power analysis is always speculative - you have to guess what the reality is. A power analysis is NOT data analysis. It sounds like you've already conducted the study. So the original power analysis cannot tell you ANYTHING about the results. A "new" power analysis could tell you how to optimize the NEXT study, but cannot inform the current study. All your results are represented by your statisitcal analysis (your parameter estimates, confidence intervals, and p-values). There have been many articles in the statistical literature on why post hoc power analysis makes no sense and can only lead to contradictions. Uninformed researchers often resort to this to try to explain why their results were NOT statistically significant - but this is not possible. If your results WERE statistically significant, well that's your results (you can't argue with statistical significance). Search for the paper "The Abuse of Power: the Pervasive Fallacy of Power Calculations as Data Analysis"- http://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf pu blished in The American Statistician. It's sad when reviewers aske researches to do ridiculous things that make no sense statistically.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I don't think power analysis is applicable to cluster analysis - there are no hypotheses being tested, and no statistical significance. $\endgroup$
    – Jeremy Miles
    Commented Jun 3, 2015 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.