Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Without citing sources, Wikipedia defines the cross-entropy of discrete distributions $P$ and $Q$ to be

\begin{align} \mathrm{H}^{\times}(P; Q) &= -\sum_x p(x)\, \log q(x). \end{align}

Who was first to start using this quantity? And who invented this term? I looked in:

J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," Information Theory, IEEE Transactions on, vol. 26, no. 1, pp. 26-37, Jan. 1980.

I followed their introduction to

A. Wehrl, "General properties of entropy," Reviews of Modern Physics, vol. 50, no. 2, pp. 221-260, Apr. 1978.

who never uses the term.

Neither does

S. Kullback and R. Leibler, "On information and sufficiency," The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79-86, 1951.

I looked in

T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, 2006.


I. Good, "Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables," The Annals of Mathematical Statistics, vol. 34, no. 3, pp. 911-934, 1963.

but both papers define cross-entropy to be synonymous with KL-divergence.

The original paper

C. E. Shannon, "A Mathematical Theory of Communication," Bell system technical journal, vol. 27, 1948.

Doesn't mention cross entropy (and has a strange definition of "relative entropy": "The ratio of the entropy of a source to the maximum value it could have while still restricted to the same symbols").

Finally, I looked in some old books and papers by Tribus.

Does anyone know what the equation above is called, and who invented it or has a nice presentation of it?

share|improve this question

It seems to be closely related to the concept of Kullback–Leibler divergence (see Kullback and Leibler, 1951). In their article Kullback and Leibler discuss the mean information for discriminating between two hypotheses (defined as $I_{1:2}(E)$ in eqs. $2.2-2.4$) and cite pp. 18-19 of Shannon and Weaver's The Mathematical Theory of Communication (1949) and p. 76 of Wiener's Cybernetics (1948).


Additional aliases include the Kullback-Leibler information measure, the relative information measure, cross-entropy, I-divergence and Kerridge inaccuracy.

share|improve this answer
Thanks! I checked those references, but I am still having trouble finding the term "cross entropy" or a matching equation. Please let me know if you saw one in one of the articles or books. – Neil G Jul 10 '12 at 8:06
You can also search backwards in Google scholar for articles with different aliases published up to a certain year (e.g., cross-entropy up to 1980). – Itamar Jul 10 '12 at 8:37
+1: Thanks for the tip. – Neil G Jul 10 '12 at 8:50
Regarding your recent edit, I am interested in the history of the form given in my question. I already noticed that early papers were using "cross entropy" to mean "KL divergence". (Note that the Kullback paper is in my question.) – Neil G Jul 10 '12 at 16:59
Sorry, I missed the Kullback paper in the question – Itamar Jul 10 '12 at 17:19
up vote 3 down vote accepted

Thanks to @Itamar's suggestion, I found a mention in:

I. J. Good, "Some Terminology and Notation in Information Theory," Proceedings of the IEE - Part C: Monographs, vol. 103, no. 3, pp. 200-204, Mar. 1956.

It would still be really useful for me to find a nice presentation of cross-entropy.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.