In Bayesian statistics, data is considered nonrandom but can have a probability or be conditioned on. How?

Question

In Bayesian statistics, parameters are said to be random variables while data are said to be nonrandom. Yet if we look at the Bayesian updating formula $$ p(\theta|y)=\frac{p(\theta)p(y|\theta)}{p(y)}, $$ we find probability (density or mass) conditioned on the data as well as the conditional and unconditional probability (density or mass) of the data itself.

How does it make sense to consider probability (density or mass) conditioned on a constant or probability (density or mass) of a constant?

Answer 1

The Bayesian approach to (parametric) statistical inference starts from a statistical model, ie a family of parametrised distributions, $$X\sim F_\theta,\qquad\theta\in\Theta$$ and it introduces a supplementary probability distribution on the parameter $$\theta\sim\pi(\theta)$$ The posterior distribution on $\theta$ is thus defined as the conditional distribution of $\theta$ conditional on $X=x$, the observed data. This construction clearly relies on the assumption that the data is a realisation of a random variable with a well-defined distribution. It would otherwise be impossible to define a conditional distribution like the posterior, since there would be no random variable to condition upon.

The possible confusion may stem from the fact that a difference between Bayesian and frequentist approaches is that frequentist procedures are evaluated and compared based on their frequency properties, ie by averaging over all possible realisations, instead of conditional on the actual realisation, as the Bayesian approach does. For instance, the frequentist risk of a procedure $\delta$ for a loss function $L(\theta,d)$ is $$R(\theta,\delta) = \mathbb E_\theta[L(\theta,\delta(X))]$$ while the Bayesian posterior loss of a procedure $\delta$ for the prior $\pi$ is $$\rho(\delta(x),\pi) = \mathbb E^\pi[L(\theta,\delta(x))|X=x]$$

Answer 2

Maybe the confusion comes from the short hand $p(\theta|y)$ which actually means $p(\theta|Y=y)$, the random variable $Y$ interpreted as generating the data takes the fixed value $y$, fixed after actually having observed the data? So the data are random in the sense of having a distribution as long as they're uncertain, i.e., not fully observed, and then they become fixed by observation. (Nothing particularly Bayesian about this, though.)

Reading a comment on the original question, "To a subjective Bayesian, nothing is random" - nothing is really/objectively random (to a subjective Bayesian at least), however it can be random in the sense of being modelled by a random variable. So another source of confusion may be mixing up the use of the term "random" in a "philosophical" manner (referring to something that is "truly random", in the sense of having randomness as intrinsic property), and in a mathematical/technical manner, referring to something that appears as random variable in a probability model.

Answer 3

Be very careful with the statement you choose. "nonrandom" is very different from "observed".

In Bayesian statistics everything is a random variable, the only difference between these random variables is some are observed and some are hidden.

For example in your case $y$ is an observed random variable and $\theta$ is a hidden random variable, your goal is to estimate the posterior distribution of $\theta$ conditioned on the observed $y$.

That says in Bayesian mindset we shouldn't reat $y$ like a constant as in the traditional sense, instead, it's an instance, or reliazation, of an random variable. (The observed values of the variables are also called "evidence" in most of the Bayesian statistics literatures.)

Answer 4

To be concrete, consider the simple case of throwing a dice. Every face has a probability to be thrown. The outcome of all throws is non-random (it is a fixed pattern determined by throwing the dice a lot of times).
On this pattern, you can apply a new chance of appearing. If you throw with two dices, a new pattern will emerge. This is because different dices will produce different outcomes (only in the case of perfect dices, the chance distribution of each one is the same as for the others). The chance that the dices used in separate throws is the same is very small. But there is a chance. And this chance is measured by applying the chance to the non-random chance distributions of a dice (for every dice this is a different distribution, though they are all quite alike).