I've recently came across topic known as PAC-Bayesian, but I cannot find a source to read about it. Any article that I came across are talking about its application in a specific area but there is no introduction to what it exactly is.
Here are a few quick Google hits...
- PAC-Bayes Analysis: Background and Applications
- Probably Approximately Correct Learning and Vapnik-Chervonenkis Dimension
- Probably approximately correct learning on Wikipedia
- Overview of the Probably Approximately Correct (PAC) Learning Framework
From this last one, a quote:
A more refined, Bayesian extension of the PAC model is explored in . Using the Bayesian approach involves assuming a prior distribution over possible target concepts as well as training instances. Given these distributions, the average error of the hypothesis as a function of training sample size, and even as a function of the particular training sample, can be defined. Also, $1 - \delta$ confidence intervals like those in the PAC model can be defined as well.
 $=$ W. Buntine, A Theory of Learning Classification Rules. PhD thesis, University of Technology, Sydney, 1990.
I'm lately interested in this topic myself, and have been looking for some good sources as well. The most interesting one I found so far is the overview/tutorial paper by David McAllester titled A PAC-Bayesian Tutorial with A Dropout Bound.