Good introduction into different kinds of entropy I'm looking for a book or online resource that explains different kinds of entropy such as Sample Entropy and Shannon Entropy and their advantages and disadvantages.
Can someone point me in the right direction?
 A: These lecture notes on information theory by O. Johnson contain a good introduction to different kinds of entropy.
A: If your interested in the mathematical statistic around entropy, you may consult this book 
http://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf
it is freely available ! 
A: Grünwald and Dawid's paper Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory discuss generalisations of the traditional notion of entropy. Given a loss, its associated entropy function is the mapping from a distribution to the minimal achievable expected loss for that distribution. The usual entropy function is the generalised entropy associated with the log loss. Other choices of losses yield different entropy such as the Rényi entropy.
A: The entropy is only one (as a concept) -- the amount of information needed to describe some system; there are only many its generalizations. Sample entropy is only some entropy-like descriptor used in heart rate analysis.
A: Jaynes shows how to derive Shannon's entropy from basic principles in his book.
One idea is that if you approximate $n!$ by $n^n$, entropy is the rewriting of the following quantity
$$\frac{1}{n}\log \frac{n!}{(n p_1)!\cdots (n p_d)!}$$ 
The quantity inside the log is the number of different length n observation sequences over $d$ outcomes that are matched by distribution $p$, so it's a kind of a measure of explanatory power of the distribution.
A: Cover and Thomas's book Elements of Information Theory is a good source on entropy and its applications, although I don't know that it addresses exactly the issues you have in mind.
