Understanding parameter as a random variable in Bayesian statistics

Question

If I understand correctly, in Bayesian statistics, a parameter is a random variable. When estimating the parameter, a prior distribution is combined with the data to yield a posterior distribution.

Question:

1. Is every data point (in the sample as well as the population) generated by the same realization of the parameter?
2. If yes, why would I care about the distribution (i.e. other possible realizations and their respective probability masses or probability density values) of the parameter? After all, I am trying to find out something about this particular population from the sample and the prior.
3. If not, how is this reflected in the formulas of Bayesian parameter estimation, if at all?

At the same time, I understand that my beliefs, whether initial (reflected by the prior) or updated (reflected by the posterior) come as a distribution, and I have no problem with that. But I wonder if/why I should assume the parameter itself is a random variable.

Edit: I have received a couple of answers which are helpful, but I would appreciate another one or a few that are more to the point.

Answer 1

In frequentist statistics, we assume we don't know anything about the parameter(s) of interest prior to seeing the data.

In Bayesian statistics, we relax this assumption by acknowledging we might know something about the possible values of the parameter(s) of interest prior to seeing the data - for example, how likely it is that the parameter might take on values in a certain range. After we see the data, we can refine/update this prior knowledge. So the prior distribution(s) of the parameters(s) of interest encapsulate our knowledge/belief about the possible values of this(these) parameters(s) prior to seeing the data, while the corresponding posterior distributions refine this knowledge/belief with input from the data.

Both frequentist and Bayesian statistics assume a parameter is an unknown quantity which must be estimated. But they diverge in terms of the inputs required for estimating that quantity: data only for frequentist statistics versus prior knowledge/belief plus data for Bayesian statistics.

When Bayesians report the mean/median/mode of a posterior distribution for a specific parameter, they do so because they believe that is the most likely value of the parameter (i.e., unknown quantity to be estimated). But the parameter is unknown, so a measure of uncertainty around this most likely value (e.g., 95% credible interval) gives a sense of where the true value of the parameter may live.

As a frequentist approaching the Bayesian field, I found it helpful to think of the parameter as something unknown we are trying to estimate and the prior and posterior distributions as tools for encapsulating our state of knowledge/belief about this unknown entity before and after we see the data, respectively.

While frequentists adopt the hard-line approach that we don't know anything about the unknown entity we are chasing after, Bayesians adopt a more realistic and nuanced approach that we may actually know something and that, once we see the data, we will improve upon that knowledge.

So we care about distributions in the Bayesian context because they are convenient vehicles for expressing the current state of knowledge/belief (for prior distributions) and the hopefully improved state of knowledge/belief achieved after seeing the data (for posterior distributions).

Addendum:

Personally, I subscribe to the view captured so eloquently by Sander Greenland in the article Bayesian perspectives for epidemiological research: I. Foundations and basic methods, International Journal of Epidemiology, Volume 35, Issue 3, 1 June 2006, Pages 765–775 (which is available at https://academic.oup.com/ije/article/35/3/765/735529):

"It is often said (incorrectly) that ‘parameters are treated as fixed by the frequentist but as random by the Bayesian’. For frequentists and Bayesians alike, the value of a parameter may have been fixed from the start or may have been generated from a physically random mechanism. In either case, both suppose it has taken on some fixed value that we would like to know. The Bayesian uses formal probability models to express personal uncertainty about that value. The ‘randomness’ in these models represents personal uncertainty about the parameter’s value; it is not a property of the parameter (although we should hope it accurately reflects properties of the mechanisms that produced the parameter)."

See http://thestatsgeek.com/2015/04/22/bayesian-inference-are-parameters-fixed-or-random/ for more musings on this topic.

Answer 2

Is every data point (in the sample as well as the population) generated by the same realization of the parameter?

• In the sample, the answer is "yes," if you are thinking of things such as heteroskedasticity as having a form like $\sigma^2_{\Delta{t}}=\Delta{t}\sigma_0^2$
• In the population, the answer could be yes if you think of the distribution as a Dirac functional, but not automatically in the population. Consider the case where a casino changes the probability it will win from time to time. For a fixed 100 experimental draws at the casino the probability of the casino winning will be the weighted average of the changing value. The timing of the structural break(s) is unknown. If the probability of the casino winning is close enough to .5 modeling a structural break is meaningless as the natural noise will swamp the reality, unless the change is stark such as going from a 50% chance of winning to a 99% chance of winning with the break at 50 draws. After the fact, the joint probability of winning is a fixed value over a fixed set of draws and fixed if the population is fixed, but not fixed if the population is still being created.

If yes, why would I care about the distribution (i.e., other possible realizations and their respective densities) of the parameter? After all, I am trying to find out something about this particular population from the sample and the prior.

Your question mirrors the Bayesian complaint that Frequentist methods violate the likelihood principle. Why consider samples unseen in performing a t-test? Because Frequentist methods average over the sample space. Why consider realizations of the parameter irrelevant to your sample? The weak response is because Bayesian methods average over the parameter space.

This question would be a very legitimate attack, particularly on subjective Bayesian methods, if one wanted to be polemical. Although the likelihood principle has been shown to be faulty under some ways of thinking about it, the philosophical construction of Frequentist methods doesn't seem to mirror the Bayesian work. That should be a paper even though it would probably share the faults of the likelihood principle in Bayesian statistics.

The stronger answer is that even though as $lim_{n\to\infty}\hat{\theta}\to\theta$ we lack an infinite sample size. As such, since Bayesian methods treat randomness as uncertainty instead of as chance as Frequentist methods do, this is a quantification of the uncertainty remaining in your understanding of nature. If you need to act on the parameter, and you have whittled it down to just two possible values $\hat{\theta}_A$ and $\hat{\theta}_B$, say with $\Pr(\theta=\hat{\theta}_A)=.75$ it may be dangerous and it is certainly incoherent (in the de Finetti sense) to ignore $\hat{\theta}_B$ as the one true possible value.

To provide a concrete example, I tested 78 models of bankruptcy and found the cumulative posterior probability for 76 of them was $1/10,000^{th}$ of one percent while the other two were approximately 54% and 46%. Fortunately, neither model shared variables. One reason to care is that, almost surely, I have the true model wrong. Model averaging of the predictive density allowed me to create a remarkably small out-of-sample error. I care about the high probability models, and if I had the resources to calculate the results under a full model averaging which included the 76 low probability models, the results would not have changed within the number of digits I consider significant.

If not, how is this reflected in the formulas of Bayesian parameter estimation, if at all?

This is Bayes theorem. If one is very intellectually honest, then the true, extreme Subjectivist view, a la Savage, requires nothing more than a proper posterior density. If I am going to gamble with you, as per de Finetti, about whether or not gravity is valid by stepping off of a ten story building, then I should want to consider alternate views of reality before making my gamble. Now if I wanted to include decision theory, since stepping off a ten-story building is a natural version of the all-or-nothing cost function, then if my beliefs against gravity are sufficiently strong, it follows that I should step off the building. In doing so, I would only worry about this one experiment as repeatability becomes an issue if I am wrong. In this case, your question has no meaning unless I am correct. On the other hand, if I am gambling money, then quadratic loss would, in most real cases, be the appropriate loss function given the nature of the demand curve for gambles and the relationship to revenues from gambles.

The potential for a changing parameter is reflected in Bayesian updating. Your question is meaningful only in repetition. That is the epitome of Bayesian updating in a purely Subjectivist framework. How do I model subjective draws of parameters, by running many experiments and joining them to narrow the posterior down to nature's solution? This is a system of thinking built around the generative model.

EDIT I think I should back up a bit. There are more than one interpretation and more than one axiomatization of Bayesian methods. They underly your questions a bit.

In the subjective interpretation, parameters are randomly drawn from a distribution. That distribution is the prior density. If you think about the numerator of Bayes theorem, $f(x|\theta)\pi(\theta)$, then it follows logically that the numerator depends on the prior strongly. Since $\theta$ is random, it follows that an experiment can be thought of as an instantiation of $\theta$. If you do another experiment, then it is another instantiation of $\theta$. The goal is to find the true distribution of the parameters. That distribution could have infinite mass on a single point and zero mass everywhere else.

In the objective interpretation, the parameters are fixed as with the Frequentist methodology, but they are unknown. The prior represents a quantification of the unknown probability that $\theta=k$. The likelihood is the distribution of the sample. There is some parameter $\theta$ which is known to nature that nature uses to create a sample $X$. Nature's prior has an infinite mass on a single point and is zero elsewhere. Your prior contains the information about what you have discovered up to this point. The likelihood only considers the sample that was seen and ignores the rest of the sample space.

There is no mathematical difference of form between the two interpretations. There is also a "convenience interpretation." It would go something like this. Bayesian methods are really useful, but figuring out priors are not. If a prior that does not injure discovery of the parameter can be created, then the most convenient and simple prior should be used since the prior can be incredibly valuable in regularizing the sample. In this view parameters are still random variables, but no one thinks much about what it means. It is just useful.

There are three main sets of axioms behind Bayesian thinking. In some cases the choice actually does matter. This is not due to calculation differences, but due to theoretical differences. For example, Savage's axioms allow researchers to separate utility and probability. de Finetti's axioms do not allow researchers to separate utility from probability. This is because probability doesn't exist in de Finetti's construction.

de Finetti has two axioms. The first is that a bookie will not accept bets that result in a sure loss in all states of nature. The second is that the bookie will accept all finite bets at the prices stated by the bookie. This is an unusual way to motivate a probability test of the speed of an object at standard temperature and pressure, but it works. It restates probability in terms of gambles. Notice that neither probability nor utility is mentioned in either axiom. Probability in de Finetti's world is just a calculation we use to think about the world and doesn't really exist. Neither does utility. Therefore if you are using utility and probability together, they are indistinguishable since they are both abstract calculations that exist to help understand the world. They are merely constructs of the mind.

As an example, consider how a Frequentist and a Bayesian would understand the game of Cho Han. To understand the Bayesian perspective, watch the 1962 Japanese movie Zatoichi. Cho Han is a game that depends on whether the dice come up even or odd. It is commonly used as a device in Yakuza movies. This is because, as any physicist, magician or con man will tell you, there is no such thing as a random dice roll or coin toss. The outcome is uncertain to the audience but perfectly certain to those in the know. It is impossible for a sample to be random because once it is done it is fixed. You know it. The question is how can parameters be random variables.

What is missing is the policy that creates the parameters. In a perfectly specified model there is no way to distinguish between a set of experiments with $\theta$ drawn from $\pi(\theta)$ and $\theta=\theta_{true}$ with uncertainty about where $\theta_{true}$ is located.

As to question two, you should read about the likelihood controversy. The likelihood principle probably isn't valid on its face, but is the Bayesian version of your question two. It is a very deep question and cannot have a shallow answer. You could write a book and certainly an article on it.

The Likelihood principle is built on two principles and Frequentist inference violates it. It is built on two principles the conditionality principle and the sufficiency principle. If the conditionality principle and the sufficiency principle hold, then p-values are always an incorrect way to determine inference. Both the conditionality principle and the likelihood principle are attractive individually to most statisticians, but jointly they could be argued to take apart Frequentism. Your question could be viewed as the Frequentist parallel.

As such, you got a deeper answer than you intended. In fact, if I were a doctoral student, I might sit down and spend time pondering your question two. There may be a deep underlying principle there.

See for example, stack exchange likelihood question

or

likelihood lecture

Answer 3

Your question: Is every data point generated by the same realization of the parameter?

My answer: No. I think the parameter changes each time a sample is drawn.

My reasoning: I looked into page 83 of "Statistical Rethinking: A Bayesian Course with Examples in R and Stan," by Richard McElreath (1st Edition), and found the following code:

sample_mu <- rnorm(1e4, 178, 20)
sample_sigma <- runif(1e4, 0, 50)
prior_h <- rnorm(1e4, sample_mu, sample_sigma)
dens(prior_h)

This code basically generates 10,000 draws from a normal distribution, where the mean follows a normal prior, and sigma follows a uniform prior. The code indicates that the parameter mu and sigma changes each time a sample is drawn.

Suppose otherwise and say the same parameter value is used for all the sampled observations. Then this sample will contain information about a single value from the distribution of the parameter, from which you cannot estimate the hyper-parameters.