Are these independent: the sample, randomized rule, and random variable having the prior distribution on the parameter space?

Question

In section 1.3 of Bickel and Doksum's Mathematical Statistics 2006, the risk function of a nonrandomized rule $d$ is the expectation of loss of the rule wrt the random sample. $$ R(\theta, d) := E_{X\sim P_\theta} l(\theta, d(X)) $$

1. the risk function of a randomized rule $\delta$ (distributed according to $\lambda$) is further taking expectation wrt the randomized rule. $$ R(\theta, \delta) := E_{\delta \sim \lambda} E_{X\sim P_\theta} l(\theta, d(X)) $$

   Q: Are the randomized rule $\delta$ and the sample supposed to be independent? I would like to know if we can exchange the order of the two expectations as: $$ R(\theta, \delta) = E_{X\sim P_\theta} E_{\delta \sim \lambda} l(\theta, \delta(X)) $$

2. In section 3.3, they studied the relation between minimax rule and Bayes rule in terms of some two-player 0-1 game, where they introduced the nature and a statistician as the two players. The nature chooses a prior distribution $\pi$ over the parameter space $\Theta$, and the statistician chooses a randomized rule. The payoff one player pays to the other is

   $$ r(\pi, \delta) := E_{\theta \sim \pi} R(\theta, \delta) = E_{\theta \sim \pi} E_{\delta \sim \lambda} E_{X\sim P_\theta} l(\theta, \delta(X)) $$

   Q: Are $\theta$, the randomized rule $\delta$ and the sample supposed to be independent? I would like to know if we can exchange the order of the three expectations as:

   $$ r(\pi, \delta) = E_{\delta \sim \lambda} E_{\theta \sim \pi} E_{X\sim P_\theta} l(\theta, \delta(X)) $$

3. The reason of my question in 2 is:

   They said that given $\delta$, $\arg\max_\pi r(\pi, \delta)$, when exists, can be chosen to the point mass probability measure on $\arg\max_\theta R(\theta, \delta)$. It means that, given a randomized rule $\delta$, the least favorite prior distribution $\pi_\delta$ on $\Theta$, can chosen to be a single $\theta_\delta$, i.e. we can remove the randomization of $\theta$

   Q: I would like to know if it is also similarly true that given $\pi$, the Bayes randomized rule $\arg\min_\delta r(\pi, \delta)$, when exists, can be chosen to have the point mass probability measure on $\arg\min_d E_{\theta \sim \pi} E_{X\sim P_\theta} l(\theta, d(X)) $?

   If yes, it implies a Bayes randomized rule is always a Bayes nonrandomized rule. But I am not sure if this conclusion is correct.

   I think that this can be true, if we can exchange the order of $ E_{\delta \sim \lambda}$ and $ E_{\theta \sim \pi}$ in $r(\pi, \delta) $ as in part 2, which is why I ask if $\theta$ and $\delta$ are independent in part 2.

Thanks.

The relevant pages are here for section 1.3 and here for section 3.3.

Answer 1

1. Are the randomized rule δ and the sample supposed to be independent?

There is no reason to restrict randomised rules in this manner. For instance, randomisation may only occur for some values of the observation/sample.

2. Are θ, the randomized rule δ and the sample supposed to be independent?

In a joint Bayesian model, sample and parameter are dependent since the sampling distribution is the conditional distribution. While the randomised rule cannot depend on the unknown $\theta$ in a functional manner, since it can [functionally] depend on the sample, it possibly does [probabilistically] depend on the unknown $\theta$.

3. a Bayes randomized rule is always a Bayes nonrandomized rule

This is incorrect: A randomised Bayes rule may differ from a non-randomised one and achieve better performances from a risk perspective. For instance, in finite parameter spaces with strictly convex losses, a minimax rule is always a randomised Bayes rule but not necessarily a non-randomised Bayes rule