2
$\begingroup$

I'm performing visualization of a dataset clustered with k-means. I compute a weight for each cluster and I draw a circle as big as its weight. But it seems like after the clustering some values are too high respect to the data set. For example the biggest weight is 1117797 while the smallest is just 2.75, I've performed normalization between [0,1] and the visualization due to that dissimilarity is not good.

Should I normalize data in a different way?, I've read about z-score but I'm not sure how to apply it in order to give less relevance to this big clusters.

Additional info: Average: 16213 Standard deviation: 110985.9

The problem I'm solving: I have around 500k text comments and they are represented in a vector space model using a Term frequency – Inverse document frequency matrix. In the end every document is represented as a vector where each dimension represents the weight of a term in the corpus.

Edit:

So far I've got the following: I consider a cluster an "outlier" of the data if its weight is greater than the average plus the standard deviation multiplied by 3. Formally is an outlier if satisfies:

$w_{i} >= \mu + \sigma * 3$

Then I do a scaling in [0,1] of the set of "outliers" and then multiply every element by $a * max_{w_{j} < \mu + \sigma * 3}(w_{j})$. It is, take the max of the "non outliers" and use it as a baseline to put another weight to the outlier points such that they remain bigger but not too much big. I've set $a = 1.7$ because it gives me nice graphical results but I am not sure this "experimental" method would work for different types of data (which may be the case in my problem).

Edit2:

Based on the Anony-Mousse suggestion I used the log function to smooth the differences between the weights, this gave some interesting results because indeed the clusters does not have such huge differences but they seem to have a very similiar size between them, I've added images to make it more clear:

Without any data standarization:

Without any data standarization

With log scale:

Log scale

As Anony-Mousse suggested this is a "Zipfian" scenario but the log scale seems to smooth way to much the differences between the weigts, for example the big blue cluster should be much more big but allowing the rest of the clusters to be seen.

|cite|improve this question
$\endgroup$
  • $\begingroup$ What do you mean by the "weight for each cluster"? $\endgroup$ – roundsquare Jul 24 '16 at 2:58
  • $\begingroup$ Its the "importance" of the cluster, as every document is a vector I take the sum of every vector in a cluster and then sum all the components, that gaves a sense of the cluster "importance" that is what I call the weight. $\endgroup$ – bones.felipe Jul 24 '16 at 3:26
  • $\begingroup$ I see. Why is that your measure of importance? It basically means that documents with more words are more important... is that really true for what you are doing? I.e. with a term-frequency matrix, the "importance" is the number of words in each document. This seems problematic since there are often cluster that pick up "small" documents i.e. whose vector is closer to the origin and others that pick up "large" documents i.e. those further from origin. I think something like the number of documents would scale better and give you a better idea of the importance of a cluster. $\endgroup$ – roundsquare Jul 24 '16 at 22:01
  • $\begingroup$ Not exactly, I am not doing only the term frequency matrix but adding Inverse document frequency such that kind of effect you are mentioning is neutralized, but indeed at some point the size of the documents impose some bias. $\endgroup$ – bones.felipe Jul 24 '16 at 23:56
1
$\begingroup$

You are probably seeing a Zipfian behavior.

Your values are non-negative, and stddev >>> mean.

Have you tried using log(weight) or sqrt(weight)?

|cite|improve this answer
$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. $\endgroup$ – gung♦ Jul 26 '16 at 1:21
  • $\begingroup$ Sometimes, I believe short answers are better. They are more likely to be read, rather than scanned for keywords and images. $\endgroup$ – Anony-Mousse Jul 26 '16 at 6:29
  • $\begingroup$ @Anony-Mousse I've updated the question with the result of your suggestion, and yes, this seems to be the typical zipfian scenario but the smothing seems to be way to high. $\endgroup$ – bones.felipe Jul 28 '16 at 0:19
  • $\begingroup$ You can also try sqrt. In particular if you use disks, the area of a disk is radius^2, so sqrt is even required for an accurate representation. $\endgroup$ – Anony-Mousse Jul 28 '16 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.