For a long time I had a bit of difficulty understanding what "degree of belief" means.

Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about this already.

Basically I think about some algorithm that generates data. Our task is to figure out what is this algorithm given some data we observe.

So prior is the information that we have so far have about the algorithm.

Let's say that there are 100 different possible algorithms and one of them is the data generating algorithm. Our task is to figure out which algorithm has generated the data that we observed.

From prior observations we can exclude 75 algorithms because they are not compatible with the data that has so far been observed.

So in this sense, the prior is a representation of the already observed data. Sort of like an entropy/compression. If the prior is saying that we excluded 99 algorithms already then it means that the data that has been seen so far is compressed into the prior perfectly.

This is a thought experiment, and I would like to read more about this interpretation of bayesian statistics. I wonder, is there some literature about this ?

In other words, I would like to understand the bayesian view in terms of algorithms/information theory/compression/entropy/turing machines/ergodicity.

I like the idea that entropy is a measure of the lack of information about a "system", but instead of system, I would rather say "algorithm" that generates data.

There are two sources where uncertainty can come from:

1) We cannot measure every degree of freedom (internal state) of the algorithm.

2) The data we have observed is finite and it is not enough to pin down precisely which algorithm was generating the data we observed.

So these are my intuitive understandings about bayesian view. Is there some literature about this already? Does this have some name? Where can I read about this interpretation more?


Studies in Subjective Probability edited by Kyburg and Smokler. It is out of print but begins with Venn's discussion of this and covers the topic.

The Foundations of Statistics by Leonard Jimmie Savage, though painfully technical in its math if memory serves me. It is in print. It is like reading a microeconomics text, again, if memory serves me.

Cox, R. T. (1961). The Algebra of Probable Inference. Baltimore, MD: Johns Hopkins University Press. I think it is out of print, but I have a copy. It approaches this issue through sentential logic, which in turn becomes a way to frame belief.

  • $\begingroup$ Thank you, interesting points towards directions I can explore ! Have not been "there" yet. $\endgroup$
    – user70990
    Jan 13 '19 at 3:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy