# Understanding the entropy of a set

I have a limited statistics background, so I will try to be as specific as possible.

I'm reading a paper on a method to fingerprint mobile devices using their accelerometer. To demonstrate that the method is robust, the author measures parameters from 16.000 devices; I don't know if it's relevant, but the parameters are the Offset (O) and the Sensibility (S) of the sensor.

He then filter them by keeping only the values of devices which has been measured exactly twice. For every device, he calculates the "distance" between every O, and then between every S. He finds out that

The 95th percentile for O distance is 0.045. For S this distance is 0.0037.

First question: what does that mean? Moreover, what is the 95th percentile, and why does he use it?

He then uses these 95th percentile distances to create a scatter plot of O-S, dividing it into blocks of equal size and counting the data points in each block (Cxy). From this he is able to calculate overall entropy of the distribution (which he does not define) as follows: Second question: what does the entropy tells me? I know it's a measure of disorder in a system, so I assume that it says something like "there are no clusters of similar informations, all the informations are disordered in the graph". But, how much? Shouldn't I compare this value with something?

The paragraph is then concluded saying that

Small variations of the grid origin had minimal effects on the entropy estimate (specifically, we saw less than 0.01 bit of entropy difference between the smallest and largest estimated value, 7.493 vs. 7.502). We consider this to be a confirmation that the result is robust.

Sorry for the long quote, but I really can't figure out what does that mean: what does it tell me if entropy has a small variation between the smaller and the largest estimated value? That's the third question.

You can find the paper here: Paper

• This question seems to be too broad. It also is confusing. Please make it clear what source(s) you are referring to? – Michael R. Chernick Jan 16 '17 at 12:31
• I'm sorry for it being confusing. I added a link to the paper at the bottom of the thread. – Alessandro Jan 16 '17 at 12:43
• The question seems clear enough. The first thing to note is that the authors are coming at their analysis from an information theoretic perspective. This field has developed since the original Shannon-Weaver papers and framework in the post-WWII era and, while it has overlap with statistics, is really a separate discipline. Next, entropy is an I-T measure of global uncertainty and, as such, is a different metric from one that would support their claim of "unique" identification of a mobile handset. Clearly given your confusion, it's not all that helpful. – Mike Hunter Jan 16 '17 at 14:27
• Finding a better metric is more of a statistical question concerning accuracy or rate of misclassification. Finally, using the 95th quantile (or whatever) is a purely subjective choice on the part of the authors of this paper. As with the statistical significance of p-values at the 0.05 level, it is, at best, a rule of thumb or convention with no theoretical motivation. – Mike Hunter Jan 16 '17 at 14:29

First question: What does that mean? Moreover, what is the 95th percentile, and why does he use it?

The 95th percentile for O distance is 0.045. For Sz this distance is 0.0037.

The x-th percentile is the value which is higher than x% of all values in a dataset. Per definition, the median is the 50-percentile. The percentile can be used to described a distribution. One reason to use the 95th-percentile could be to disregard outliers - those with the highest 5% distance.

Second question: what does the entropy tell me?

Entropy describes a distribution and can be well visualized and understood as a property of a histogram.

The entropy is zero, if only a single event (here: distance value) occurs - therefore there is no uncertainty. The histogram has only a single bin. If all events occur with equal probability (or if working with empirical data: occurred equally often), the entropy is at its maximum.

As the entropy depends on the number of possible events, it can be normalized such as dividing by the maximum entropy possible. Doing so, the entropy can only be a value in the interval of [0,1].

In this specific paper, the data is plotted as a scatterplot and the scatterplot area is discretized in equally sized sub-areas to form a histogram by counting the occurrences in each area. An area is hereby defined as an event and the amount of occurrences inside are counted.

Third question: what does it tells me if entropy has a small variation between the smaller and the largest estimated value?

Now the center of the grid - which originated from using equally sized areas for all values below the 95th percentile - depends on the data (!). By varying the grid center (maximum and minimum as possible alternative centers) and checking the change of the entropy, the author tries to estimates the robustness of his approach. If a small change of the grid center would result in a large change of the entropy, then his approach would be sensitive to the input data.

I assume there are plenty other options to estimate the robustness. I specifically wonder why the author does not estimate robustness of the identification as opposed to the robustness of entropy depending on the grid center.