# Measures of entropy/information: distinguish clustered configurations that would have the same information entropy

Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D distribution over a square grid and suppose to consider two configurations (i.e. two distributions over these square grid ) which have the same entropy. I would like to know if there exist modified measures of entropy/information which take into account also clustering so that a clustered configuration of my system is no more degenerate with respect to a more sparse one.

• Although it's not perfectly clear what you want, your request sounds remarkably like this question, which has answers: stats.stackexchange.com/questions/17109/….
– whuber
Commented Jun 11, 2013 at 20:35
• the problem is that I do not have enough reputation score to post an image since I am newcomer of this forum, otherwise it would be very simple to explain what I am looking for by this picture. Commented Jun 11, 2013 at 21:36
• If you post a picture somewhere on the Web, you can supply a link to it.
– whuber
Commented Jun 11, 2013 at 21:41
• link Commented Jun 11, 2013 at 21:48
• I would like to have a measure according to which the two configurations are not degenerate and in particular a measure in which clustered configurations have a lower entropy than sparse ones as shown in the figure. Commented Jun 11, 2013 at 21:50

Given that this measure (and all similar measures) reduce a complete data set to a single value, pretty much any not perfectly correlated second measure will help distinguishing such situations.

For example, you could rotate your coordinate system by 45° and then compute entropy on the rotated grid. Maybe one can construct a configuration that has the same entropy in the first measure, but not when rotated 45°.

+--+--+    +--+--+
|  |  |    |  | O|
|  |O |    |  |  |
+--+--+    +--+--+
| O|  |    |  |  |
|  |  |    |O |  |
+--+--+    +--+--+

Nearby objects may then end up in the same cell when using a different grid - or not:

X   X   X      X   X   X
\ / \ / \      \ / \O/ \
X  OX   X      X   X   X
/ \O/ \ /      / \ / \ /
X   X   X      X  OX   X
\ / \ / \      \ / \ / \

An even simpler example is just to vary the grid size!

+---+---+---+     +----+----+
|   |   |   |     |    |    |
|   |   |   |     |    |    |
+---+---+---+     |    |O   |
|   |  O|   |     +----+----+
|   |O  |   |     |   O|    |
+---+---+---+     |    |    |
|   |   |   |     |    |    |
|   |   |   |     +----+----+
+---+---+---+
• Thank you for your answer but entropy should be invariant under the rotation of the system that you propose and in general under permutation/rearrangement of the cells. $H(p1,p2,p3)= H(p2,p3,p1) = H(p2,p1,p3)$ and etc, even if I rotate the system the $p_i$ which enter in the sum defining the entropy are the same, therefore I would find the same result. Commented Jun 12, 2013 at 8:33
• Your grid will change due to the rotation, and I'm not aware of a continuous notion of entropy. Commented Jun 12, 2013 at 8:50
• Although it's not perfectly clear to me which rotation you are proposing, in my case given the fact that I don't have many observations for building the empirical distribution on which I calculate the entropy, I fear that the rotation will produce an almost equal situation, however I will try. Thank you. Commented Jun 12, 2013 at 9:01
• why continuous notion of entropy? I don't get the point with this observation. Commented Jun 12, 2013 at 9:02
• Grids cause artifacts. If you change the grid, your get different artifacts. Commented Jun 12, 2013 at 9:24