3
$\begingroup$

I have subdivided the globe into 100 sq mile bins and then collected how many tweets were sent with a geolocation within each bin. At first I color coded each bin on the map based off a standardized value of (tweetInBin)/(maxTweetInAnyBin). This produced only one "hot spot" with all the other places being almost uniform in color.

http://i.stack.imgur.com/4gJms.png (Sadly I don't have enough rep yet to actually embed the photo)

As you can see from the percentile chart, the 100th percentile is so large in population relative to the other bins, that standardizing by the max val destroys any meaningful representation.

My question to you guys is how do I color code my data so that I can have a meaningful heat map. I was toying with the idea of linearizing the data by color coding based off of percentiles instead of based off of population. Basically the 100th percentile would get 100% intensity on the color scale, while the 50th percentile would get 50% intensity on the color scale and so on. My only gripe with this method is that it marginalizes the actual difference in populations much like taking the median marginalizes the outliers.

How do I handle the outliers while still conveying population information?

$\endgroup$
  • 2
    $\begingroup$ Are you aware of the ColorBrewer site and package? Lots of good ideas there. $\endgroup$ – Peter Flom May 16 '13 at 19:02
  • $\begingroup$ As you are dealing with count data you might try taking the square root as an alternative to your rank-ordering transformation. $\endgroup$ – Peter Ellis May 16 '13 at 19:35
  • 1
    $\begingroup$ I would take the natural log instead of the square root. This allows information about very small values as well as very large ones to be clearly communicated. For exponential functions the natural log transforms to a form that is usually accessible via polynomial fitting. $\endgroup$ – EngrStudent May 16 '13 at 22:46
  • 1
    $\begingroup$ @EngrStudent: I think you should post that as an answer. $\endgroup$ – naught101 May 17 '13 at 1:51
  • $\begingroup$ @naught101 - okay. :) $\endgroup$ – EngrStudent May 17 '13 at 17:39
5
$\begingroup$

I believe in this kind of situations it's more important to ask "what do I need to show?" rather than "what should the picture look like?" There are times when map with a sea of white and pink and a couple of big red dots being very useful; there are also times that the same design can lead to biased decision. It all depends on what do you mean by meaningful.

If your intention is to show off the extreme, then I don't see why you need to transform anything. If you would also like the audience to see the less extreme, then a better way is to break the data into 10 or so chunks with equal group sizes (aka decile). For lay audience, transformation, regardless up or down; exponential or logarithmic, is often a difficult concept. It's so much easier to perceive "this color represents the top 10% of the tweet density."

Still, the root of the skewness has not been solved. If you believe that the top is skewed so badly because they have more people, then adjust for the people by showing # of tweets divided by population divided by area (or phone users/tweeter accounts in that area if you're resourceful enough to get those data). I feel that would really tell a better story if people living closer together tend to tweet more. Otherwise, it's just another apparent conclusion: human activities are more frequent at places where a lot of human beings gather.

$\endgroup$
  • $\begingroup$ Upvoted for the remark "human activities are more frequent at places where a lot of human beings gather." I think that could be the abstract for most contemporary research. $\endgroup$ – Sycorax May 17 '13 at 13:18
  • 1
    $\begingroup$ @DJE, you might like this XKCD comic $\endgroup$ – xan May 17 '13 at 13:52
  • $\begingroup$ I think Tweet/PopDensity sounds like a much better story to tell. I don't know if you ever read XKCD but I think that this comic strip sums everything up. Haha! I thought about standardizing by population somehow, but now you just solidified the idea in my mind. Thanks. $\endgroup$ – SlightlyCyborg May 17 '13 at 13:57
2
$\begingroup$

You can transform the data, of course, to make it more linear, but then you're changing the visual interpretation so make sure the interpretation is still meaningful. If you want to treat the one high value as an outlier, given it a qualitatively different color.

For instance, make the high value black and put the others on a normal sequential color map. For your example, the color legend might say black = 101 and white-to-blue = 0-to-50. There would still be a lot of indistinguishable whitish values, but that's OK because presumably the numbers aren't significantly different either.

If there are several high values, you can put them on a separate sequential color map. For instance, 0-50/50-100 = white-blue/red-black.

EDIT: adding pics of colored exponential data.

Since color sequential:

enter image description here

Dual color sequential (I suppose the cut point doesn't have to be in the middle):

enter image description here

The range of perception is increased (you can now distinguish Wyoming from Idaho) at the cost of introducing the categorical break between the high and lows (you can no longer quantitatively compare Alaska with California).

$\endgroup$
  • $\begingroup$ Added pics, @Penguin_Knight. Not sure what the proper name of this scale is or if it's what you understood from my description. I wouldn't call it "diverging", except in the sense that it's like a diverging scale with one part flipped, which may be what you meant. $\endgroup$ – xan May 17 '13 at 13:50
  • $\begingroup$ Oh, I see. Yes, a bit different from what I thought, more like a dual scale maybe? But it does the job. $\endgroup$ – Penguin_Knight May 17 '13 at 14:21
1
$\begingroup$

I would take the natural log instead of the square root. This allows information about very small values as well as very large ones to be clearly communicated. For exponential functions the natural log transforms to a form that is usually accessible via polynomial fitting.

I don't like your graph. It is almost, but not quite a proper eCDF. Empirical Cumulative Distribution plots (eCDF) are very informative, though there are many graphical methods for exploratory data analysis.

Even better is to try a few different fits, and try to make some analytic/symbolic sense of the list of numbers.

So I approximated your data using a table:

table of values

I then used the MatLab curve fitting tool and determined that a power fit was a nice fit model. (This took ~2 seconds so don't consider it rigorous, more of a demonstration of a method.

Plot of Fit and error

The parameters of the fit are:

Numeric Result

I am sure you realize how wonderful and appropriate this is. The power law fits really well even though I have crappy, crudely approximated data. It fits the underlying model well and is informative in useful and interesting ways, if you know how to take it apart. (link, link, link)

So now, you have an analytic form, $ y = 1-Ax^b$, that you can invert or otherwise use to inform transformations that more approximately linearly inform your color mapping. You have some links that can inform your understanding of the meaning of your fit as well.

Best of luck.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.