# Simple technique to identify number of clusters in dataset

I have a survey app (programmed using Ruby On Rails), and I am required to cluster the responses.

I am using a Ruby library called AI4R and my code (in the event it is useful...) looks like the below (example code filched from AI4R)

# 5 Questions on a post training survey
questions = [   "The material covered was appropriate for someone with my level of knowledge of the subject.",
"The material was presented in a clear and logical fashion",
"There was sufficient time in the session to cover the material that was presented",
"The instructor was respectful of students",
"The instructor provided good examples"]

# Answers to each question go from 1 (bad) to 5 (excellent)
# The answers array has an element per survey complemented.
# Each survey completed is in turn an array with the answer of each question.
[ 1, 2, 3, 2, 2],   # Answers of person 1
[ 5, 5, 3, 2, 2],   # Answers of person 2
]

data_set = DataSet.new(:data_items => answers, :data_labels => questions)

# Let's group answers in 4 groups
clusterer = Diana.new.build(data_set, 4)


This in turn lets me create graphs like this (the survey has questions which are linked to themes/axes).

The problem is that right now you have to pick (read guess) the number of clusters to pass into AI4R.

I saw on Wikipedia that there is a technique called the Elbow Method (illustrative picture from Wikipedia),

which compares the number of clusters with the variance that they explain. This technique would be perfect for my needs, but I don't know how to implement it in Ruby (or with pen and paper).

What stats technique can I use to calculate the number of clusters vs the percentage variance that they explain?

• – chl Oct 2 '13 at 13:12
• "Variance explained" is the between-cluster variance. It is all about ANOVA. Perform one-way ANOVA by factor = cluster variable, for every feature (question). Sum SSerror across the features (this is W). Sum SStotal across the features (this is T). Variance explained is (T-W)/(k-1) where k is the number of clusters. – ttnphns Oct 2 '13 at 13:36
• Do you have a prior on the distribution of the number of clusters? If so you could just pick the number of clusters with the maximum likelihood. – Mimshot Oct 2 '13 at 14:16