For a long time I had a bit of difficulty understanding what "degree of belief" means.
Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about this already.
Basically I think about some algorithm that generates data. Our task is to figure out what is this algorithm given some data we observe.
So prior is the information that we have so far have about the algorithm.
Let's say that there are 100 different possible algorithms and one of them is the data generating algorithm. Our task is to figure out which algorithm has generated the data that we observed.
From prior observations we can exclude 75 algorithms because they are not compatible with the data that has so far been observed.
So in this sense, the prior is a representation of the already observed data. Sort of like an entropy/compression. If the prior is saying that we excluded 99 algorithms already then it means that the data that has been seen so far is compressed into the prior perfectly.
This is a thought experiment, and I would like to read more about this interpretation of bayesian statistics. I wonder, is there some literature about this ?
In other words, I would like to understand the bayesian view in terms of algorithms/information theory/compression/entropy/turing machines/ergodicity.
I like the idea that entropy is a measure of the lack of information about a "system", but instead of system, I would rather say "algorithm" that generates data.
There are two sources where uncertainty can come from:
1) We cannot measure every degree of freedom (internal state) of the algorithm.
2) The data we have observed is finite and it is not enough to pin down precisely which algorithm was generating the data we observed.
So these are my intuitive understandings about bayesian view. Is there some literature about this already? Does this have some name? Where can I read about this interpretation more?