Consider PAC-Bayesian bounds used in learning theory (as defined in say section $1.2$, page $3$ of this paper, https://arxiv.org/pdf/1707.09564.pdf).
I want to know what is the precise mathematical expression that is asked to be true when one says that the "prior distribution has to be independent of the training data". Can someone kindly state that as an equation in probability theory?
This is confusing because to make sense of a notion of "independence" here one has to somehow first be able to imagine the "prior" and the "data" both as random variables on the same probability space. To the best of my knowledge the only formal notion of independence is that of between $2$ random variables both of which are mapping from the same probability space.
If say one had to make a list of $K$ priors and allow oneself to choose one of these priors then there is a way to rewrite such bounds with an extra $log(K)$ term in the numerator under the square-root on the RHS.
Now lets say that I do some auxiliary experiments with training this predictor on certain samples and realize that there is a certain prior which gives me a good RHS then I guess I am not allowed to use this prior in the bound because this is now dependent on the data. But,
(a) Am I allowed to include this prior in the list of $K$ prior options? Or will that also break the assumptions of this proof?
(b) Am I allowed to use this prior if I say use it in a PAC-Bayes bound for the same predictor but I have say changed the changed the data distribution from which the samples are being taken (from the auxiliary experiments to the case when the bound is being evaluated) or I have thrown away from the support of the distribution the part of the data on which I did these auxiliary experiments.