0
$\begingroup$

I need to calculate the entropy of 100 instances of 5 sensor signals in python. The sensor values take real numbers. After doing some literature search, I suppose I need to compute joint differential entropy. For this, I should estimate a multivariate probability density function defining my sensor values. Since I do not have a sound knowledge on information theory, I cannot validate my thoughts. Please guide me how to achieve this.

P.S. I am looking for theoretical suggestions and not related to coding.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ multivariate joint PDF on 100 data points will by garbage $\endgroup$
    – Aksakal
    Commented Mar 26, 2018 at 18:35
  • $\begingroup$ @Aksakal lol. Any suggestions on how to tackle this problem? any alternatives? $\endgroup$
    – Ijjz
    Commented Mar 26, 2018 at 18:39

1 Answer 1

Reset to default
1
$\begingroup$

This is how it's been with a lot of data and sensors. "Entropy Based Anomaly Detection Applied to Space Shuttle Main Engines." A. Agogino and K. Tumer. In Proceedings of the IEEE Aerospace Conference, Big Sky, MO, March 2006.

See Eq.2: $$H(S)=\sum_{i=1}^n \frac {n_i} n\log_2\frac{n_i} n$$ They bin the data, and don't need to estimate the multivariate probability. In their case with 147 sesnors and megabytes of data it would be a garbage joint density, as it would be in your with 5 sensors but only 100 observations.

You could say that they're using some form of a empirical distribution function. You could try Kernel density as others do, but with 100 observations it'll still be garbage :)

Improve this answer
$\endgroup$
4
  • $\begingroup$ The paper certainly put me on the right track. I should be looking for entropy estimators like nearest neighbor based, etc. I need a clarification on this paper though. If I have understood it well, the entropy is min when all the sensor values fall under similar bins(high or low valued). This might not be always true. For me one sensor should be high and the other low in a 'normal' or 'predictable' scenario. Is this assumption valid? or have I misunderstood it? $\endgroup$
    – Ijjz
    Commented Mar 27, 2018 at 20:31
  • $\begingroup$ I'm not sure I understood your question. Take a look at my answer to an earlier question: there's more uncertainty in a uniform distribution than in a bell shaped one. Pay attention to signs in Shannon's definition and Agogino's. $\endgroup$
    – Aksakal
    Commented Mar 27, 2018 at 20:52
  • $\begingroup$ I have understood the concept of entropy. I am confused with the binning method in the paper. The author's interpretation of low entropy indeed suggests that most of the values at one timestamp are at similar levels of discreteness(i.e. not uniformly distributed). Doesn't it also imply that the signals are all high or all low to obtain low entropy? Isn't it a hard constraint on the sensor values? $\endgroup$
    – Ijjz
    Commented Mar 27, 2018 at 21:12
  • $\begingroup$ The bin sizes are equal, which is important. So, the uniform distribution will produce the highest Shannon entropy (most uncertainty), but they dropped the negative sign from usual equation. is this the confusion? $\endgroup$
    – Aksakal
    Commented Mar 27, 2018 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.