40
$\begingroup$

In Bayesian data analysis, parameters are treated as random variables. This stems from the Bayesian subjective conceptualization of probability. But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'

It seems like the obvious answer is 'yes', because then trying to estimate the parameter would almost be nonsensical. An academic citation for this answer would be greatly appreciated.

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Give me that parameter and I will define a distribution for it. :-) $\endgroup$ – Anne van Rossum Jan 29 '14 at 15:44
43
$\begingroup$

IMHO "yes"! Here is one of my favorite quotes by Greenland (2006: 767):

It is often said (incorrectly) that ‘parameters are treated as fixed by the frequentist but as random by the Bayesian’. For frequentists and Bayesians alike, the value of a parameter may have been fixed from the start or may have been generated from a physically random mechanism. In either case, both suppose it has taken on some fixed value that we would like to know. The Bayesian uses formal probability models to express personal uncertainty about that value. The ‘randomness’ in these models represents personal uncertainty about the parameter’s value; it is not a property of the parameter (although we should hope it accurately reflects properties of the mechanisms that produced the parameter).

Greenland, S. (2006). Bayesian perspectives for epidemiological research: I. Foundations and basic methods. International Journal of Epidemiology, 35(3), 765–774.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Being slightly contrarian, couldn't one argue that the (position, momentum) of a particle is a "parameter" that we might try to estimate? One might argue there is no "fixed" value of this parameter, and that we should genuinely regard it as a distribution. Regarding unknowns specifically as distributions rather than fixed values seems to be what nature does in certain situations. I don't think this reasoning is very appealing to a Bayesian in practice, but I think to fully answer OPs question some discussion of the nature of randomness is needed. $\endgroup$ – guy Jan 29 '14 at 16:41
  • 3
    $\begingroup$ I don't see what is 'uncongenial' about considering a distribution as the target of inference. Indeed, indexing things with parameters is anyway optional e.g. one may put assert one's uncertainty using distributions over functions directly (see Neal and Williams on Gaussian Processes). And one need not have any particular view of 'randomness' to represent uncertainty with the probability calculus. Inference based on sampling (theory) arguably does need such a theory, but as far as I can see Bayesianism isn't (or at least does not need to be.) $\endgroup$ – conjugateprior Jan 29 '14 at 19:43
  • 2
    $\begingroup$ I don't agree that a Bayesian assumes that a parameter has a fixed value and it's only their personal uncertainty that drives them to define the parameter as a distribution rather than a point. I've tried to expand on that in my answer. Your personal/theoretical uncertainty is part of the distribution, but it seems to me that your model is essentially averaging across variables that are left out of the model and that creates a distribution, even if your personal priors are very precise. $\endgroup$ – Wayne Jan 30 '14 at 3:36
29
$\begingroup$

The Bayesian conception of a probability is not necessarily subjective (c.f. Jaynes). The important distinction here is that the Bayesian attempts to determine his/her state of knowledge regarding the value of the parameter by combining a prior distribution for its plausible value with the likelihood which summarises the information contained in some observations. Hence, as a Bayesian, I'd say that I am happy with the idea that the parameter has a true value, which is not known exactly, and the purpose of a posterior distribution is to summarise what I do know about its plausible values, based on my prior assumptions and the observations.

Now, when I make a model, the model is not reality. So in some cases the parameter in question does exist in reality (e.g. the average weight of a wombat) and in some questions it doesn't (e.g. the true value of a regression parameter - the regression model is only a model of the outcome of the physical laws that govern the system, which may not actually be captured fully by the regression model). So to say that there is one true fixed parameter value in the real world is not necessarily true.

On the flip side, I would suggest that most frequentists would say there is one true value for the statistic, but they don't know what it is either, but they have estimators for it and confidence intervals on their estimates which (in a sense) quantifies their uncertainty regarding the plausibility of different values (but the frequentist conception of a probability prevent them from expressing this as directly).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I always thought "subjective probabilities" were called subjective because they refer to a property of the subject doing the calculation (i.e. her knowledge) rather than a property of the objective reality (e.g. the weight distribution of a not-perfectly-fair dice). $\endgroup$ – nikie Jan 30 '14 at 7:07
  • 1
    $\begingroup$ @nikie there are principles, such as MaxEnt and transformation groups for defining priors in an objective way. In that case the computation gives the same answer regardless of who or what is performing the calculation (a robot would come to the same conclusion). Frequentist statistics is not without subjectivity either, for instance how to set $\alpha$ for hypothesis tests, but the intention there was to remove the subjectivity from statistics in the sense of "personal probability/belief" rather than "degree of plausibility" (IIRC). $\endgroup$ – Dikran Marsupial Jan 30 '14 at 7:59
  • $\begingroup$ I know, but they're still subjective probabilities, right? Because they're still describing the subject's knowledge about some parameter (which, for a frequentist, wouldn't be a random variable at all) $\endgroup$ – nikie Jan 30 '14 at 8:30
  • $\begingroup$ There isn't necessarily a subject. A set of robots or computers could all perform the same calculation and arrive at the same conclusion, whether using a frequentist or an objectivist Bayesian approach. It is the state of knowledge, regardless of the subject performing the calculation, which is why it is objective rather than subjective. $\endgroup$ – Dikran Marsupial Jan 30 '14 at 11:38
17
$\begingroup$

To your main point, in Bayesian Data Analysis (3rd ed., 93), Gelman also writes

From the perspective of Bayesian data analysis, we can often interpret classical point estimates as exact or approximate posterior summaries based on some implicit full probability model. In the limit of large sample size, in fact, we can use asymptotic theory to construct a theoretical Bayesian justification for classical maximum likelihood inference.

So perhaps it's not Bayesians who should "admit" that there are, in truth, single real parameter values, but frequentists who should appeal to Bayesian statistics to justify their estimation procedures! (I say this with tongue firmly in cheek.)

As an aside, I object to the blanket statement that Bayesian statistics is premised on subjective probability, and implication that Bayes is subjective while other inferential paradigms are not. That is certainly one argument that can be posed, perhaps also including the perspective of the "coherence of bets" argument, but see Gelman who here defines "Bayesian" as a statistician that uses the posterior distribution $\Pr(\theta|y)$, and here where he argues against overly restrictive definitions.

But the idea that there are single parameters in nature or in social systems is just a simplifying assumption. There might be some ornate process generating observable results, but discovering that system is incredibly complicated; supposing that there is a single fixed parameter value simplifies the problem dramatically. I think that this cuts to the core of your question: Bayesians shouldn't have to "admit" to making this simplification any more than Frequentists should.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Would you mind elaborating on why you reject Bayesian inference being based on subjective probability? The introductory texts I've read (Kruschke, Lynch) all seem to frame it that way. Is it that it is only partially subjective (coming from the prior)? $\endgroup$ – ATJ Jan 29 '14 at 14:30
  • $\begingroup$ @ATJ I hope that this clarifies my point. There are other arguments that one could advance, but the real sticking point for me was the implicit assumption that Bayesian stats is subjective in ways that other paradigms are not. For example, I would argue against the characterization in Bernd's quote because it seems just as "personal" to favor a method of unbiased point estimation over a framework of posterior variability. $\endgroup$ – Sycorax says Reinstate Monica Jan 29 '14 at 15:02
  • $\begingroup$ @ATJ, introductory texts tell a story to motivate methods. Something like that story may have originally motivated the methods. But that doesn't mean that that story has much influence on the assumptions that people make in applying those methods in practice. (And the story may be nonsense: For example, the idea that the probabilities involved in statistics can be defined in terms of frequencies in the way that introductory frequentist texts sometimes say doesn't make sense--Google Alan Hajek's "15 Arguments" papers. That doesn't mean that frequentist statistics doesn't work; it has.) $\endgroup$ – Mars Jan 31 '14 at 5:50
9
$\begingroup$

Do you think that there is a single "true fixed parameter" for something like the contribution of milk drinking to a child's growth? Or for the decrease in a tumor's size based on the amount of chemical X you inject into a patient's body? Pick any model you're familiar with and ask yourself if you actually believe that there is one true, universal, precise and fixed value for each parameter, even in theory.

Ignore measurement error, just look at your model as if all measurements were perfectly accurate and infinitely precise. Given your model, do you think that each parameter realistically has a specific point value?

The fact that you have a model indicates that you are leaving some details out. Your model will have an amount of imprecision because you're averaging over the parameters/variables that you've left out in order to make a model -- a simplified representation of reality. (Just as you don't make a 1:1 map of the planet, complete with all details, but rather a 1:10000000 map, or some such simplification. The map is a model.)

Given that you're averaging across the left-out variables, the parameters for the variables you include in your model will be distributions, not point values.

That's only part of the Bayesian philosophy -- I'm ignoring theoretical uncertainty, measurement uncertainty, priors, etc -- but it seems to me that the idea that your parameters have distributions makes intuitive sense, in the same way that descriptive statistics have a distribution.

|cite|improve this answer
$\endgroup$
7
$\begingroup$

But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'

In my opinion, the answer is yes. There is an unknown value $\theta_0$ of the parameter and the prior distribution describes our knowledge/uncertainty about it. In the Bayesian mathematical modelling, $\theta_0$ is considered as the realization of a random variable following the prior distribution.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ What exactly is $\theta_0$, that is, what does that subscript $0$ represent? $\endgroup$ – nbro Nov 16 at 4:02
2
$\begingroup$

If we go and couple Bayesianism with a deterministic universe (before you say anything with the word 'quantum' in it, humour me and recall that this is not physics.stackexchange) we get some interesting results.

Making our assumptions explicit:

  1. We have a Bayesian agent being part of and observing a deterministic universe.
  2. The agent has limited computational resources.

Now, the deterministic universe may be one where atoms are newtonian little billiard balls. It may be entirely non-quantum. Let's say it is.

The agent now flips a fair coin. Think about that for a second, what does a fair coin constitute in a deterministic universe? A coin that has a 50/50 probability ratio?

But it is deterministic! With enough computing power you can calculate exactly how the coin will land, purely by simulating a model of a coin being flipped in the same manner.

In a deterministic universe a fair coin would be a disc of metal with uniform density. No force compels it to spend more time with one face down than the other (think about how weighted dice function.)

So the agent flips a fair coin. Yet, the agent is not quite powerful enough. It does not have sharp enough eyes to measure how the coin spins when flipped, it sees but a blur.

And so it says "This coin will land a heads with 50% probability." Lack of information leads to probabilities.

We may look at the phase space of how a coin is thrown. A large multidimensional coordinate system with axes pertaining to direction of throw, force of throw, spin of the coin, speed and direction of wind and so on. A single point in this space corresponds to a single possible coinflip.

If we ask the agent from before to colour in the coordinate system with a greyscale gradient corresponding to the agent's assignment of probability of heads for every given throw, it will most colour it all a uniform shade of grey.

If we the gradually give it more powerful internal computers with which to compute probabilities of heads, it will be able to make more and more discerning colourings. When we finally give it the most powerful internal computer, making it omniscient, it will effectively paint a strange checkerboard.

Fair coins are not made of probabilities, they are made of metal. Probabilities exist only in computational structures. So says the Bayesian.

|cite|improve this answer
$\endgroup$
-3
$\begingroup$

There are improper priors, for example Jeffreys, which has a certain relation to Fishers Information matrix. Then it is not subjective.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Could you elaborate on how Jeffreys's prior and its relation to the Fisher information matrix means that Bayesian inference is not subjective? As I understand it, the main reason to use Jeffreys's prior is that it is invariant to alternative parameterizations of the model. Additionally, in a multidimensional setting, these Jeffreys's prior can become highly informative, and the results are controversial (Gelman, BDA 3, p. 53). Does this undermine its 'objectivity'? $\endgroup$ – Sycorax says Reinstate Monica Jan 29 '14 at 12:57
  • $\begingroup$ @user777, since it based on parameters of the density on hand it is objective. Suppose I multiply likelihood by 1, do I have then subjetive prior for the probability? Since posterior probability is related to prior x likelihood. $\endgroup$ – Analyst Jan 31 '14 at 9:58
  • $\begingroup$ And frequentist also have to invoke Axiom of The True Model if they want to use likelihood... :) $\endgroup$ – Analyst Jan 31 '14 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.