Today I ran across the book "Information theory: A tutorial introduction" by James Stone and thought for a moment or two about the extent of use of information theory in applied data science (if you're not comfortable with this still somewhat fuzzy term, think data analysis, which IMHO data science is a glorified version of). I'm well aware of the significant use of information theory-based approaches, methods and measures, especially entropy, under the hood of various statistical techniques and data analysis methods.

However, I'm curious about the extent/level of knowledge that is needed for an applied social scientist to successfully select and apply those concepts, measures and tools without diving too deep into mathematical origins of the theory. I look forward to your answers, which might address my concern within the context of the above-mentioned book (or other similar books - feel free to recommend) or in general.

I would also appreciate some recommendations for print or online sources that discuss information theory and its concepts, approaches, methods and measures in the context of (in comparison with) other (more) traditional statistical approaches (frequentist and Bayesian).

  • 2
    $\begingroup$ Perhaps one of the most known and "applied" case of use of entropy takes place when building up a tree. One of the possibilities when algorithm splits is to take the information gain metric, which is the difference between entropy between the top level and the down level. You have more information here en.wikipedia.org/wiki/Information_gain_in_decision_trees $\endgroup$
    – D.Castro
    Apr 7, 2015 at 6:46
  • $\begingroup$ @D.Castro: Thank you for your comment - I'm aware of that case (and even posted an answer on this exact topic either here on Cross Validated, or on Data Science SE site). I'm hoping for a more comprehensive coverage/discussion of the subject matter. $\endgroup$ Apr 7, 2015 at 6:49
  • 1
    $\begingroup$ For me, and in large part, it's a matter of the discipline or field in which one is trained as well as the geographic continent. In my view, physicists, mathematicians and practitioners of pure machine learning are much more likely to receive in-depth exposure to information theory than are, say, statisticians, economists or quantitative financial analysts. In addition, I would double down on this for people trained in Europe, i.e., Europeans are much more likely to be familiar with IT. However, the advent of models for statistical learning is changing that for data scientists in the States. $\endgroup$ Dec 14, 2016 at 12:57
  • $\begingroup$ @DJohnson Minutest of minute points but in Britain and perhaps elsewhere IT == information technology. Otherwise your impressions resemble mine. $\endgroup$
    – Nick Cox
    Dec 14, 2016 at 14:06
  • $\begingroup$ @NickCox Thanks, your point holds for the States as well. It was a longish comment and, space permitting, I would have spelled the words out or, better yet, have introduced the meaning of the acronym at an earlier point. $\endgroup$ Dec 14, 2016 at 14:45

1 Answer 1


So the first part of question: Do data scientists need to know information theory? I thought the answer is no until very recently. The reason I changed my mind is one crucial component: noise.

Many machine learning models (both stochastic or not) use noise as part of their encoding and transformation process and in many of these models, you need to infer the probability which the noise affected after decoding the transformed output of the model. I think that this is a core part of information theory. Not only that, in deep learning, KL divergence is a very important measure used that also comes from Information Theory.

Second part of the question: I think the best source is David MacKay's Information Theory, Inference and Learning Algorithms. He starts with Information Theory and takes those ideas into both inference and even neural networks. The Pdf is free on Dave's website and the lectures are online which are great

  • 3
    $\begingroup$ It's an excellent book. Anyone interested should also glance at en.wikipedia.org/wiki/David_J._C._MacKay $\endgroup$
    – Nick Cox
    Dec 14, 2016 at 12:40
  • $\begingroup$ Thank you for your answer (+1 and potential accept, if no more comprehensive answers will pop up soon enough). Special appreciation for the references. I'm surprised you ran across this almost forgotten, but important, question of mine. :-) $\endgroup$ Dec 15, 2016 at 2:47
  • $\begingroup$ Yeah it's interesting. You should never give up on a question. Came to me after I attended NIPS2016 and I saw all those talks on KL divergence and noise impact to encoders. $\endgroup$
    – Ambodi
    Dec 15, 2016 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.