Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D distribution over a square grid and suppose to consider two configurations (i.e. two distributions over these square grid ) which have the same entropy. I would like to know if there exist modified measures of entropy/information which take into account also clustering so that a clustered configuration of my system is no more degenerate with respect to a more sparse one.
8
-
$\begingroup$ Although it's not perfectly clear what you want, your request sounds remarkably like this question, which has answers: stats.stackexchange.com/questions/17109/…. $\endgroup$– whuber ♦Commented Jun 11, 2013 at 20:35
-
$\begingroup$ the problem is that I do not have enough reputation score to post an image since I am newcomer of this forum, otherwise it would be very simple to explain what I am looking for by this picture. $\endgroup$– user1234383Commented Jun 11, 2013 at 21:36
-
$\begingroup$ If you post a picture somewhere on the Web, you can supply a link to it. $\endgroup$– whuber ♦Commented Jun 11, 2013 at 21:41
-
$\begingroup$ link $\endgroup$– user1234383Commented Jun 11, 2013 at 21:48
-
$\begingroup$ I would like to have a measure according to which the two configurations are not degenerate and in particular a measure in which clustered configurations have a lower entropy than sparse ones as shown in the figure. $\endgroup$– user1234383Commented Jun 11, 2013 at 21:50
|
Show 3 more comments
1 Answer
$\begingroup$
$\endgroup$
6
Given that this measure (and all similar measures) reduce a complete data set to a single value, pretty much any not perfectly correlated second measure will help distinguishing such situations.
For example, you could rotate your coordinate system by 45° and then compute entropy on the rotated grid. Maybe one can construct a configuration that has the same entropy in the first measure, but not when rotated 45°.
+--+--+ +--+--+
| | | | | O|
| |O | | | |
+--+--+ +--+--+
| O| | | | |
| | | |O | |
+--+--+ +--+--+
Nearby objects may then end up in the same cell when using a different grid - or not:
X X X X X X
\ / \ / \ \ / \O/ \
X OX X X X X
/ \O/ \ / / \ / \ /
X X X X OX X
\ / \ / \ \ / \ / \
An even simpler example is just to vary the grid size!
+---+---+---+ +----+----+
| | | | | | |
| | | | | | |
+---+---+---+ | |O |
| | O| | +----+----+
| |O | | | O| |
+---+---+---+ | | |
| | | | | | |
| | | | +----+----+
+---+---+---+
answered Jun 11, 2013 at 20:07
-
$\begingroup$ Thank you for your answer but entropy should be invariant under the rotation of the system that you propose and in general under permutation/rearrangement of the cells. $H(p1,p2,p3)= H(p2,p3,p1) = H(p2,p1,p3)$ and etc, even if I rotate the system the $p_i$ which enter in the sum defining the entropy are the same, therefore I would find the same result. $\endgroup$ Commented Jun 12, 2013 at 8:33
-
$\begingroup$ Your grid will change due to the rotation, and I'm not aware of a continuous notion of entropy. $\endgroup$ Commented Jun 12, 2013 at 8:50
-
$\begingroup$ Although it's not perfectly clear to me which rotation you are proposing, in my case given the fact that I don't have many observations for building the empirical distribution on which I calculate the entropy, I fear that the rotation will produce an almost equal situation, however I will try. Thank you. $\endgroup$ Commented Jun 12, 2013 at 9:01
-
$\begingroup$ why continuous notion of entropy? I don't get the point with this observation. $\endgroup$ Commented Jun 12, 2013 at 9:02
-
$\begingroup$ Grids cause artifacts. If you change the grid, your get different artifacts. $\endgroup$ Commented Jun 12, 2013 at 9:24