How is data generated in the Bayesian framework and what is the nature on the parameter that generates the data?

Question

I was trying to re-learn Bayesian statistics (every time I thought I finally got it, something else pops out that I didn't consider earlier....) but it wasn't clear (to me) what the data generation process in the Bayesian Framework actually is.

The frequentist framework its clear to me. There is some "true" parameter(s) $\theta$ and that parameter generates the data according to the distribution that it parametrizes.

However, in the Bayesian setting, we model the parameter as a random variable. That part does not confuse me. It makes sense, because a Bayesian interprets this probability as the uncertainty in its own beliefs. They are ok with assigning a probability to nonrepeatable events. So the way that I interpreted "Bayesianism" was that, it believe that there is some parameter generating the data, it definitively is unknown but nevertheless, fixed once it was decided by "nature" (and maybe nature did decide randomly what it was supposed to be). Nevertheless, it is fixed and hence it creation was a "nonrepeatable event". Even though it was nonrepeatable, we are only trying to update our own belief of $\theta$ given data. Therefore, the data might have been generated by any of the parameters under consideration by our probability distribution (prior), but nevertheless, the parameter is fixed and unknown. We are just attaching a probability value to it.

With this view, it makes sense to me to assume that the data generation process is nearly identical to the frequentist one. "Nature" selects the parameter $\theta$ using the "true" "prior" distribution $P^*(\theta)$ and once the random variable takes its "true" (but fixed) realization, it starts generating the data that we observe.

Is this the standard way to interpret the data generation process for in the Bayesian framework?

The main thing about my view is that, the parameter $\theta$ is definitively fixed (viewed as a realization of a r.v.), and it generates the data according to $\theta$. Hence, another very important point on my view is that, for me, that our prior is only a quantifiable way of expressing our uncertainty on the fixed (and nonrepeatable) event of creating the parameter $\theta$. Is that how people interpret the prior $P(\theta)$?

---

Side humorous note:

I wish could just ask "Nature" how she is doing it and settle this once and for all ... lol.

Answer 1

It is pretty straightforward: there are no differences between Bayesians and frequentists regarding the idea of the data-generating model.

To understand this, consider first that the data-generating model is mathematically encoded in the likelihood, which is the basis for the inference of Bayesians and frequentists alike. And there is zero difference between a Bayesian and frequentist likelihood.

Now, you could say: that doesn't mean that Bayesians think that the parameters of the data-generating process are fixed. Sure, but really, it makes very little sense to think otherwise - what would be the point of estimating a quantity that is not fixed? What would that even mean mathematically? Of course, it could be that you have a quantity that is not a value, but a distribution. But then you estimate the distribution, so it is fixed again.

The real difference, as @Xi'an says, is not in the assumption about how our data is generated, but in the inference. So, when you say

However, in the Bayesian setting, we model the parameter as a random variable.

I would disagree - we model our knowledge / uncertainty about the true parameter as a random variable - that is the subtle, but important difference - we treat the parameter as random variables to explore our uncertainty about their "true" value.

Answer 2

Pages 3 and 4 of BDA by Gelman et al., 3rd ed., are illuminating. Bayesian statistics aims to make inference from data using probability models for observables and unobservable quantities. We refer to the unobservable quantities as parameters, even if the distinction is not always clear-cut. In Bayesian statistics all uncertainty about the variables involved in the model is represented using probability. Thus we need to setup a full probability model, i.e., a joint probability among all variables involved in our problem, both observable and unobservable, i.e., parameters. This means that we use random variables to represent both. It doesn't mean that we believe the parameter to be random: it means simply that our knowledge of the real value of the parameters is limited, and we represent whatever limited knowledge we have before observing data through the prior probability distribution. We then observe data and condition on the observed data using a model for the data generating process (which gives rise to a certain likelihood function) and Bayes' rule, to obtain a posterior probability distribution, which quantifies the remaining uncertainty in our knowledge about the unobservable quantities.

In other words we use random variables for parameters not because we believe that there are no true parameters, but because we have a limited knowledge of them, which improves after observing data for the measurable variables, but it doesn't disappear completely. As a matter of fact, there are technical conditions under which the posterior distribution tends to a Dirac delta (thus the random variable used to represent the parameter becomes degenerate) in the limit for the number of observations which goes to 0. If there was no "true" value for the parameter, this wouldn't make a lot of sense. Now, surely these conditions are not always valid, but in many standard Bayesian analyses (even if not all) we do not doubt the existence of a true model, and of true or fixed values for the unobservables.

Answer 3

Is this the standard way to interpret the data generation process for in the Bayesian framework?

No, this is not the standard interpretation. In fact, you have already recognised in your question the "subjective" interpretation of probability, which is the standard basis of Bayesian statistics. Under the "subjectivist" interpretation (more properly called the "epistemic" interpretation), the prior and posterior probability distributions for the parameters are used to represent the user's uncertainty about the unknown parameters in the model. Under this account there is no assumption of any corresponding metaphysical process occurring in nature, or any randomness in nature. Indeed, under this view the Bayesian paradigm does not provide any theory at all on the "data generation process" of nature; it merely gives us a mathematical way to model our uncertainty about things in nature, and hence form an inferential and predictive theory.

Your latter description is an example of the propensity theory of probability, which posits that there is a metaphysical process that occurs in nature that is analogous to the probability calculus. This interpretation of probability assumes that there is some inbuilt metaphysical "propensity" in nature for outcomes to occur at random according to the laws of probability. As with most Bayesians, I have always found the propensity accounts to be a bit silly. It is really an example of human beings' propensity to project our own modes of thinking onto nature, and assume that there are analogues in nature to our epistemological methods and constructs. (As such, the "propensity interpretation" is more properly a propensity theory of human beings than one of probability!)

Now, you might decide to adopt the subjectivist interpretation of probability, or you might disagree with me and decide to adopt the propensity interpretation. Regardless, you are going to get yourself into an awful mess if you equivocate between these two different interpretations. That is probably what is giving you difficulties at the moment.

Answer 4

The parameter $\theta$ can only be regarded as fixed but unknown if you assume that the underlying model that you are working with is a perfect representation of the true system. However, since nature is usually much more complex than any mathematical model that we use, this assumption cannot be made. Therefore, there is no 'one true fixed' parameter of your model.

Mathematically, as you add more and more data, you will converge to a certain parameter $\theta$. However, this is due to the insufficiency of your assumptions in the modeling process. You should be careful to call it the true fixed parameter of the underlying system. Even if a parameter in your model has a physical meaning - it is only an assumption that the posterior parameter retains this interpretation completely.

The data in a Bayesian view is generated by the 'true system' - which you will never be able to model correctly. Therefore, an underlying true parameter of your assumed model cannot exist.