In Bayesian data analysis, parameters are treated as random variables. This stems from the Bayesian subjective conceptualization of probability. But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'

It seems like the obvious answer is 'yes', because then trying to estimate the parameter would almost be nonsensical. An academic citation for this answer would be greatly appreciated.

  • 4
    $\begingroup$ Give me that parameter and I will define a distribution for it. :-) $\endgroup$ Jan 29, 2014 at 15:44

8 Answers 8


IMHO "yes"! Here is one of my favorite quotes by Greenland (2006: 767):

It is often said (incorrectly) that ‘parameters are treated as fixed by the frequentist but as random by the Bayesian’. For frequentists and Bayesians alike, the value of a parameter may have been fixed from the start or may have been generated from a physically random mechanism. In either case, both suppose it has taken on some fixed value that we would like to know. The Bayesian uses formal probability models to express personal uncertainty about that value. The ‘randomness’ in these models represents personal uncertainty about the parameter’s value; it is not a property of the parameter (although we should hope it accurately reflects properties of the mechanisms that produced the parameter).

Greenland, S. (2006). Bayesian perspectives for epidemiological research: I. Foundations and basic methods. International Journal of Epidemiology, 35(3), 765–774.

  • 2
    $\begingroup$ Being slightly contrarian, couldn't one argue that the (position, momentum) of a particle is a "parameter" that we might try to estimate? One might argue there is no "fixed" value of this parameter, and that we should genuinely regard it as a distribution. Regarding unknowns specifically as distributions rather than fixed values seems to be what nature does in certain situations. I don't think this reasoning is very appealing to a Bayesian in practice, but I think to fully answer OPs question some discussion of the nature of randomness is needed. $\endgroup$
    – guy
    Jan 29, 2014 at 16:41
  • 3
    $\begingroup$ I don't see what is 'uncongenial' about considering a distribution as the target of inference. Indeed, indexing things with parameters is anyway optional e.g. one may put assert one's uncertainty using distributions over functions directly (see Neal and Williams on Gaussian Processes). And one need not have any particular view of 'randomness' to represent uncertainty with the probability calculus. Inference based on sampling (theory) arguably does need such a theory, but as far as I can see Bayesianism isn't (or at least does not need to be.) $\endgroup$ Jan 29, 2014 at 19:43
  • 2
    $\begingroup$ I don't agree that a Bayesian assumes that a parameter has a fixed value and it's only their personal uncertainty that drives them to define the parameter as a distribution rather than a point. I've tried to expand on that in my answer. Your personal/theoretical uncertainty is part of the distribution, but it seems to me that your model is essentially averaging across variables that are left out of the model and that creates a distribution, even if your personal priors are very precise. $\endgroup$
    – Wayne
    Jan 30, 2014 at 3:36

The Bayesian conception of a probability is not necessarily subjective (c.f. Jaynes). The important distinction here is that the Bayesian attempts to determine his/her state of knowledge regarding the value of the parameter by combining a prior distribution for its plausible value with the likelihood which summarises the information contained in some observations. Hence, as a Bayesian, I'd say that I am happy with the idea that the parameter has a true value, which is not known exactly, and the purpose of a posterior distribution is to summarise what I do know about its plausible values, based on my prior assumptions and the observations.

Now, when I make a model, the model is not reality. So in some cases the parameter in question does exist in reality (e.g. the average weight of a wombat) and in some questions it doesn't (e.g. the true value of a regression parameter - the regression model is only a model of the outcome of the physical laws that govern the system, which may not actually be captured fully by the regression model). So to say that there is one true fixed parameter value in the real world is not necessarily true.

On the flip side, I would suggest that most frequentists would say there is one true value for the statistic, but they don't know what it is either, but they have estimators for it and confidence intervals on their estimates which (in a sense) quantifies their uncertainty regarding the plausibility of different values (but the frequentist conception of a probability prevent them from expressing this as directly).

  • $\begingroup$ I always thought "subjective probabilities" were called subjective because they refer to a property of the subject doing the calculation (i.e. her knowledge) rather than a property of the objective reality (e.g. the weight distribution of a not-perfectly-fair dice). $\endgroup$
    – nikie
    Jan 30, 2014 at 7:07
  • 2
    $\begingroup$ @nikie there are principles, such as MaxEnt and transformation groups for defining priors in an objective way. In that case the computation gives the same answer regardless of who or what is performing the calculation (a robot would come to the same conclusion). Frequentist statistics is not without subjectivity either, for instance how to set $\alpha$ for hypothesis tests, but the intention there was to remove the subjectivity from statistics in the sense of "personal probability/belief" rather than "degree of plausibility" (IIRC). $\endgroup$ Jan 30, 2014 at 7:59
  • $\begingroup$ I know, but they're still subjective probabilities, right? Because they're still describing the subject's knowledge about some parameter (which, for a frequentist, wouldn't be a random variable at all) $\endgroup$
    – nikie
    Jan 30, 2014 at 8:30
  • 1
    $\begingroup$ There isn't necessarily a subject. A set of robots or computers could all perform the same calculation and arrive at the same conclusion, whether using a frequentist or an objectivist Bayesian approach. It is the state of knowledge, regardless of the subject performing the calculation, which is why it is objective rather than subjective. $\endgroup$ Jan 30, 2014 at 11:38

To your main point, in Bayesian Data Analysis (3rd ed., 93), Gelman also writes

From the perspective of Bayesian data analysis, we can often interpret classical point estimates as exact or approximate posterior summaries based on some implicit full probability model. In the limit of large sample size, in fact, we can use asymptotic theory to construct a theoretical Bayesian justification for classical maximum likelihood inference.

So perhaps it's not Bayesians who should "admit" that there are, in truth, single real parameter values, but frequentists who should appeal to Bayesian statistics to justify their estimation procedures! (I say this with tongue firmly in cheek.)

As an aside, I object to the blanket statement that Bayesian statistics is premised on subjective probability, and implication that Bayes is subjective while other inferential paradigms are not. That is certainly one argument that can be posed, perhaps also including the perspective of the "coherence of bets" argument, but see Gelman who here defines "Bayesian" as a statistician that uses the posterior distribution $\Pr(\theta|y)$, and here where he argues against overly restrictive definitions.

But the idea that there are single parameters in nature or in social systems is just a simplifying assumption. There might be some ornate process generating observable results, but discovering that system is incredibly complicated; supposing that there is a single fixed parameter value simplifies the problem dramatically. I think that this cuts to the core of your question: Bayesians shouldn't have to "admit" to making this simplification any more than Frequentists should.

  • $\begingroup$ Would you mind elaborating on why you reject Bayesian inference being based on subjective probability? The introductory texts I've read (Kruschke, Lynch) all seem to frame it that way. Is it that it is only partially subjective (coming from the prior)? $\endgroup$
    – ATJ
    Jan 29, 2014 at 14:30
  • $\begingroup$ @ATJ I hope that this clarifies my point. There are other arguments that one could advance, but the real sticking point for me was the implicit assumption that Bayesian stats is subjective in ways that other paradigms are not. For example, I would argue against the characterization in Bernd's quote because it seems just as "personal" to favor a method of unbiased point estimation over a framework of posterior variability. $\endgroup$
    – Sycorax
    Jan 29, 2014 at 15:02
  • $\begingroup$ @ATJ, introductory texts tell a story to motivate methods. Something like that story may have originally motivated the methods. But that doesn't mean that that story has much influence on the assumptions that people make in applying those methods in practice. (And the story may be nonsense: For example, the idea that the probabilities involved in statistics can be defined in terms of frequencies in the way that introductory frequentist texts sometimes say doesn't make sense--Google Alan Hajek's "15 Arguments" papers. That doesn't mean that frequentist statistics doesn't work; it has.) $\endgroup$
    – Mars
    Jan 31, 2014 at 5:50

Do you think that there is a single "true fixed parameter" for something like the contribution of milk drinking to a child's growth? Or for the decrease in a tumor's size based on the amount of chemical X you inject into a patient's body? Pick any model you're familiar with and ask yourself if you actually believe that there is one true, universal, precise and fixed value for each parameter, even in theory.

Ignore measurement error, just look at your model as if all measurements were perfectly accurate and infinitely precise. Given your model, do you think that each parameter realistically has a specific point value?

The fact that you have a model indicates that you are leaving some details out. Your model will have an amount of imprecision because you're averaging over the parameters/variables that you've left out in order to make a model -- a simplified representation of reality. (Just as you don't make a 1:1 map of the planet, complete with all details, but rather a 1:10000000 map, or some such simplification. The map is a model.)

Given that you're averaging across the left-out variables, the parameters for the variables you include in your model will be distributions, not point values.

That's only part of the Bayesian philosophy -- I'm ignoring theoretical uncertainty, measurement uncertainty, priors, etc -- but it seems to me that the idea that your parameters have distributions makes intuitive sense, in the same way that descriptive statistics have a distribution.

  • 1
    $\begingroup$ "your parameters have distributions" Whether parameters have distributions depends on what you mean by distribution. In reality a parameter might have a fixed value. Like for instance the number of marbles in a box, the charge of an electron, the value of a stock at a particular time. But we treat those variables with distributions (they don't have it) because the values get mixed in an ensemble of similar entities but with different values (e.g. different stocks at different dates, multiple different boxes with marbles in them, or multiple different measurements on an electron). $\endgroup$ Dec 6, 2019 at 11:51

But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'

In my opinion, the answer is yes. There is an unknown value $\theta_0$ of the parameter and the prior distribution describes our knowledge/uncertainty about it. In the Bayesian mathematical modelling, $\theta_0$ is considered as the realization of a random variable following the prior distribution.


If we go and couple Bayesianism with a deterministic universe (before you say anything with the word 'quantum' in it, humour me and recall that this is not physics.stackexchange) we get some interesting results.

Making our assumptions explicit:

  1. We have a Bayesian agent being part of and observing a deterministic universe.
  2. The agent has limited computational resources.

Now, the deterministic universe may be one where atoms are newtonian little billiard balls. It may be entirely non-quantum. Let's say it is.

The agent now flips a fair coin. Think about that for a second, what does a fair coin constitute in a deterministic universe? A coin that has a 50/50 probability ratio?

But it is deterministic! With enough computing power you can calculate exactly how the coin will land, purely by simulating a model of a coin being flipped in the same manner.

In a deterministic universe a fair coin would be a disc of metal with uniform density. No force compels it to spend more time with one face down than the other (think about how weighted dice function.)

So the agent flips a fair coin. Yet, the agent is not quite powerful enough. It does not have sharp enough eyes to measure how the coin spins when flipped, it sees but a blur.

And so it says "This coin will land a heads with 50% probability." Lack of information leads to probabilities.

We may look at the phase space of how a coin is thrown. A large multidimensional coordinate system with axes pertaining to direction of throw, force of throw, spin of the coin, speed and direction of wind and so on. A single point in this space corresponds to a single possible coinflip.

If we ask the agent from before to colour in the coordinate system with a greyscale gradient corresponding to the agent's assignment of probability of heads for every given throw, it will most colour it all a uniform shade of grey.

If we the gradually give it more powerful internal computers with which to compute probabilities of heads, it will be able to make more and more discerning colourings. When we finally give it the most powerful internal computer, making it omniscient, it will effectively paint a strange checkerboard.

Fair coins are not made of probabilities, they are made of metal. Probabilities exist only in computational structures. So says the Bayesian.


I am not sure it is a relevant question because it requires more definition than the math itself requires. Because the math itself does not require it, I am not sure asking which Bayesian interpretation is correct has a lot of meaning.

Imagine two parallel universes. They are identical in the sense that the sequence of physical events in both universes unfolds in the same way.

In other words, of the sample space $\chi$, the Universe, $U\subset\chi$ is the same in every respect, $U_1=U_2$.

Now, in Universe One, every observer believes that Nature draws fixed points, $\theta_0$, from the parameter space at the start of time, $t=0$. These fixed points are $\theta_0\subset\Theta$. An observer denoted $i$, explains their initial uncertainty about its location with a probability distribution, $\pi_i(\theta_0)$ and upon seeing the data $X_i\subset{U}$ revises their uncertainty to $\pi(\theta_0|X_i)$.

Now, in Universe Two, every observer believes that Nature draws values for parameters, $\theta_t$, from a distribution which is believed to be approximated by $\pi_i(\theta_t)$ at time $t=\tau,$ which is when observer $i$ gathers the data. The draws are random. This differs from the concept of heteroskedasticity or stationary variables. Such a person would define either in a different manner. Upon seeing the data $X_\tau\subset{U}$ at time $\tau,$ they use this additional information to improve the description of that distribution of $\theta_t$ to $\pi_i(\theta_t|X_\tau)$.

Do note that in the second case, it isn't really helpful to bring in Frequentist definitions of things like time series, heteroskedasticity, or stationary variables because their ideas are predicated on fixed points. Also, there is nothing in Universe Two that prohibits the distribution from being a Dirac Delta function. However, nothing prevents a prior in Universe One from being one either, and as such, one could completely miss $\theta_0$.

If you drop the needless subsidiary notation, you end up with $\pi(\theta|X)\propto\pi(\theta)f(X|\theta)$.

The math provides no mechanism to be able to distinguish a world with fixed but unobservable parameters and a mechanism to describe that uncertainty from a world where the parameters truly are random variables.

Which is it? Who knows?

That Frequentist methods in some sense "work," doesn't provide a solution either. There is nothing about countably additive sets that makes them better than finitely additive sets. It is true that there are use cases where only a null hypothesis method or only a Bayesian method could possibly work. They are not the general case.

Interestingly, in those handfuls of cases where only one method could be thought of as suitable, the problem isn't resolved.

For example, if the critical element of your method boils down to a sharp null hypothesis such as $$H_0:\beta_1,\beta_2,\dots\beta_k=0$$ it does depend on on a mathematical conditioning of those parameters at zero, as if it were the true fixed point. Nature isn't required to listen. Indeed, if nature were sometimes drawing parameters instead and causing false positives or negatives, the method would not be able to tell.

Likewise, if the critical element of your method is setting gambling odds, there is no way to distinguish either world. You have to use a Bayesian method, but either conceptualization will work either well.


There are improper priors, for example Jeffreys, which has a certain relation to Fishers Information matrix. Then it is not subjective.

  • 2
    $\begingroup$ Could you elaborate on how Jeffreys's prior and its relation to the Fisher information matrix means that Bayesian inference is not subjective? As I understand it, the main reason to use Jeffreys's prior is that it is invariant to alternative parameterizations of the model. Additionally, in a multidimensional setting, these Jeffreys's prior can become highly informative, and the results are controversial (Gelman, BDA 3, p. 53). Does this undermine its 'objectivity'? $\endgroup$
    – Sycorax
    Jan 29, 2014 at 12:57
  • $\begingroup$ @user777, since it based on parameters of the density on hand it is objective. Suppose I multiply likelihood by 1, do I have then subjetive prior for the probability? Since posterior probability is related to prior x likelihood. $\endgroup$
    – Analyst
    Jan 31, 2014 at 9:58
  • $\begingroup$ And frequentist also have to invoke Axiom of The True Model if they want to use likelihood... :) $\endgroup$
    – Analyst
    Jan 31, 2014 at 10:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.