1
$\begingroup$

I am trying to perform a clustering analysis for a csv file with 50k+ rows, 10 columns. I tried k-mean, hierarchical and model based clustering methods. Only k-mean works because of the large data set. However, k-mean does not show obvious differentiations between clusters. So I am wondering is there any other way to better perform clustering analysis?

The data looks like this

Revenue  Employee  Longitude Latitude  LocalEmployee BooleanQuestions ...
1000     100       xxxx      xxxx      10
...                                                                   ...

Here is part of my code:

mydata <- scale(mydata)
wss <- (nrow(mydata)-1)*sum(apply(mydata,2,var))
for(i in 2:15)wss[i]<- sum(fit=kmeans(mydata,centers=i,15)$withinss)
plot(1:15,wss,type="b",main="15 clusters",xlab="no. of cluster",ylab="with clsuter sum of squares")

fit <- kmeans(mydata,7)
clusplot(mydata, fit$cluster, color=TRUE, shade=TRUE, labels=2, lines=0)

enter image description here

$\endgroup$
1
  • $\begingroup$ Your question, as stated, may be impossible to answer. In what sense did other clustering approaches not work? What do you mean that the k-means output did not show obvious differences, are you just referring to the plot? (& is that the function from the cluster package?) The plot is a projection onto the 1st & 2nd principle component (I believe), which maximize the data variation, not the gaps between the clusters. $\endgroup$ Jul 15, 2015 at 20:46

1 Answer 1

1
$\begingroup$

Understand the algorithms you are using.

Don't treat them as black boxes!

k-means, for example, is known

  1. to be sensitive to outliers
  2. to not work well with correlations
  3. to not work well with binary variables
  4. to minimize within cluster variance

so I'm not surprised the result looks like this. You may need to more precisely figure out what your objective is. Maybe variance itself is not what you need to minimize.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.