Today I ran across the book "Information theory: A tutorial introduction" by James Stone and thought for a moment or two about the extent of use of information theory in applied data science (if you're not comfortable with this still somewhat fuzzy term, think data analysis, which IMHO data science is a glorified version of). I'm well aware of the significant use of information theory-based approaches, methods and measures, especially entropy, under the hood of various statistical techniques and data analysis methods.
However, I'm curious about the extent/level of knowledge that is needed for an applied social scientist to successfully select and apply those concepts, measures and tools without diving too deep into mathematical origins of the theory. I look forward to your answers, which might address my concern within the context of the above-mentioned book (or other similar books - feel free to recommend) or in general.
I would also appreciate some recommendations for print or online sources that discuss information theory and its concepts, approaches, methods and measures in the context of (in comparison with) other (more) traditional statistical approaches (frequentist and Bayesian).