11
$\begingroup$

It seems to me like the concepts of incorporating prior beliefs about parameters VERSUS viewing parameters as latent random variables are two VERY separate concepts, and yet I've found that they're often confounded and treated as one. Can't we have prior beliefs about what the most likely value of a parameter is without actually believing that the parameters are "random variables" i.e they do have a single "true" value, but there's just uncertainty about what that value is. The fact that we have "prior knowledge" about what the value of the parameter might be and that we encode that via a prior probability distribution doesn't seem to me to exclude the possibility that that thing we're talking about (the "parameter") is a fixed, albeit latent value.

I've taken a course in Bayesian statistics, and have read online about Bayesian analysis, but I actually still do not fully understand this.

EDIT: Just to be a bit more concrete, suppose we conduct an analysis “as a frequentist,” and infer the value of some parameter. However, because we have some prior knowledge about the parameter, we use a prior that reflects our uncertainty about the value of the parameter. But just because we used a prior doesn’t require us to assume that the parameter we’re inferring is inherently random. The resulting probability distribution of the inferred parameter is simply arising because of uncertainty, not inherent randomness in the parameter. We still believe, as frequentists, that the parameter has a fixed, latent value. So are we being Bayesian, or frequentist, or a combination of both in this type of scenario?

$\endgroup$
12
  • $\begingroup$ If you have a prior belief that you can quantify with probabilities (e.g., there's 90% probability that $X>0$) then you are effectively treating $X$ as a random variable. From a Bayesian viewpoint uncertainty and randomness are essentially synonyms. $\endgroup$
    – J. Delaney
    Commented Mar 22, 2023 at 17:56
  • $\begingroup$ To a frequentist, they are very different concepts. In a case where a rv X has a pdf depending on the value of a 'parameter' which is itself regarded as a rv with a known pdf, then a frequentist could/would use the Bayesian approach. $\endgroup$ Commented Mar 22, 2023 at 18:45
  • $\begingroup$ @J.Delaney but why not make the distinction between “being a random variable because of uncertainty” VS “being a random variable because of randomness.” It’s interesting that you mention that for a Bayesian, uncertainty and randomness are synonyms. I feel like that’s misleading and confounding separate concepts for no apparent reason (at least in my limited/possibly naive view) What are your thoughts? $\endgroup$ Commented Mar 22, 2023 at 20:28
  • 1
    $\begingroup$ What does "inherit randomness" mean? think of a coin flip. If you could measure all the relevant physical parameters (exact force applied to the coin, surrounding air density etc.) then you could perfectly predict the outcome. But when you don't have that information, you treat is a random process. So the statement $P(heads)=1/2$ really only reflects your own (subjective) uncertainty about the process, not an inherit property of it. $\endgroup$
    – J. Delaney
    Commented Mar 22, 2023 at 21:00
  • 1
    $\begingroup$ @AnIgnorantWanderer The "things that will happen in the future" are themselves physical processes that, given enough information on initial conditions, can be predicted, so that argument is just kicking the can down the road. Regarding the question in your edit, there is no coherent way of "using a prior" inside a Frequentist analysis: those are fundamentally different (and contradictory) concepts. $\endgroup$
    – J. Delaney
    Commented Mar 23, 2023 at 12:16

4 Answers 4

22
$\begingroup$

The mathematical concept of a "random variable" does not require a belief in randomness

You seem to be reading more into the mathematical concept of a "random variable" than is actually contained in the concept. Mathematically, the concept of a random variable is just a mapping from a sample space in a probability space to the real numbers, complex numbers, or some other set of values for a quantity of interest. This means that every random variable has a probability distribution and every quantity with a probability distribution is a random variable. So in fact these are not very seperate concepts; they are identical. Treated purely as a mathematical concept, this does not necessarily entail any particular metaphysical properties of the quantity, and in particular, it does not imply that the quantity is actually random in an aleatory sense. Perhaps this is a bit naughty and confusing on the part of the mathematics community, and I can see why it would lead to misunderstandings, but it arises largely for historical reasons relating to the evolution of probability theory.

The most commonly applied paradigm for Bayesian statistics treats probability as an epistemological concept that describes our own uncertainty in a quantity, but does not take a position on metaphysical issues relating to determinism, indeterminism, and randomness. There is a good treatment of the philosophical and mathematical foundations of Bayesian statistics in Bernardo and Smith (1994), which discusses the epistemological interpretation of probability in the "subjective Bayesian paradigm". This approach is agnostic on the metaphysical issue of whether or not randomness actually exists in nature --- either way we need a tool to describe our own uncertainty about quantities, and probability does this job.

$\endgroup$
5
  • 1
    $\begingroup$ When modelling a data generating process, it may often be convenient to signify whether or not we are acting as if randomness was involved. Somewhat tongue-in-cheek, should we clarify by describing the rvs as random random variables in such cases (with the alternative circumstances said to involve nonrandom random variables)? $\endgroup$ Commented Mar 23, 2023 at 19:57
  • $\begingroup$ There is generally little loss of understanding with referring to quantities as "random variables" in the usual sense, so long as it is understood that this is a mathematical setup, not a philosophical discussion pertaining to aleatory randomness. If you want to avoid this language, an alternative is just to call things "quantities" and the apply probabilities to them as needed, according to the chosen methodology. $\endgroup$
    – Ben
    Commented Mar 23, 2023 at 21:25
  • 1
    $\begingroup$ Perhaps in Bayesian analyses it does not matter whether or not randomness is involved, however the distinction is important to frequentist analyses. The issue is not mathematical (since the term rv is agnostic in this regard) nor specifically philosophical but rather one of adequately describing the statistical modelling. Perhaps instead of random random variable, to signify rvs with randomness it would be better to say X is a 'chance random variable' for clarity. $\endgroup$ Commented Mar 23, 2023 at 22:25
  • $\begingroup$ I disagree. My view is that adding new terminology of that kind would add more confusion, rather than clarifying the model. My experience is that the model can be specified clearly with standard mathematical terminology and notation. As noted in my answer, I do not think there is any distinction between RVs with and without aleatory randomness that need be made in statistical/mathematical modelling. $\endgroup$
    – Ben
    Commented Mar 24, 2023 at 1:39
  • $\begingroup$ Fair enough, that's your view. $\endgroup$ Commented Mar 24, 2023 at 2:04
5
$\begingroup$

I like to view Bayes rule intuitively as a situation where we know that some (not directly measured) unknown value B follows a known probability distribution (e.g. it is sampled from a population with a reasonably well know distribution) and there is another known value A that has a statistical relationship with the value of B.

marginal and conditional distribution

(Image from the question: Bayes' Theorem Intuition)

In this picture of Bayes rule, the parameters A and B are shown to follow a particular joint distribution. In an application of the rule, often A is measured/observed and B is unknown.

So we consider B to follow a certain (known/estimated) distribution, because it can be considered as being sampled from a larger population with a certain distribution, but that doesn't mean that B doesn't have a fixed value. It is just that this value is unknown.

Example: Say, B, could be the 'intelligence' of a person and A could be the score on an intelligence test. The joint distribution expresses that people with a same intelligence B might score differently on the intelligence test A. So people with a particular score A are not to be considered as having intelligence A, but instead they are considered to follow a posterior distribution B|A.

without actually believing that the parameters are "random variables" i.e they do have a single "true" value, but there's just uncertainty about what that value is.

Eventhough the value of B might be fixed, it is in a certain sense a random value to is, when we don't know what the value actually is. In that case we describe the value B in terms of what we do know and that is that the value B follows a particular know probability distribution.

The randomness expresses uncertainty.

  • That can be either for a single entity having variations all the time (according to some distribution).
  • Or it is a single entity having a fixed value, but being sampled from a population that fixed value is random to us as we do not know the value.

Having a fixed value, when that value is unknown, we can treat it as random. Maybe the world is deterministic or it is not, but for our analyses does it matter?

$\endgroup$
4
$\begingroup$

You can find a very good answer to your question in the Who Are The Bayesians? thread and How exactly do Bayesians define (or interpret?) probability? that discusses the specific way how Bayesian approach probability, which is one of the core concepts of the approach.

But making the answer more specific, it is not about latent variables, because there are many non-Bayesian models with latent variables. There is a whole class of latent variable models that can be treated as Bayesian models but do not have to (e.g. you can find their parameters with maximum likelihood). It's more about using priors and specific definition of probability, as you can learn from the threads mentioned above.

$\endgroup$
1
$\begingroup$

There are cases where frequentist and Bayesian inference converge on the same answer, so within these limits, one could argue that the distinctions are only philosophical.

I feel that the wording "incorporating prior beliefs" already implies that the parameter you are infering is a random variable.

  1. Your "prior belief" has to be a probability distribution rather than a fixed value, otherwise it would be a "present certainty" rather than "prior belief," and it wouldn't make sense to perform inference if you already know what the answer is.
  2. "Incorporating" implies that it needs to contribute to your answer. If I thought 1+1=3 and observe that it is 2, my original 3 is entirely discarded and not incorporated into my observation of 2. Our prior beliefs are "incorporated" via multiplication in Bayes' theorem.
  3. A probability distribution, whether multiplied by another probability distribution or a fixed value, equals another probability distribution. Hence, our answer is in the form of a probability distribution and is therefore a random variable by definition. If we insist that the answer is a fixed value in some way, how can we claim that we've multiplied in our prior beliefs as defined above?

I am having trouble understanding your "concrete scenario" because I don't know how you could actually "use a prior that reflects our uncertainty" in practice, other than via Bayes' theorem. Allow me to ignore that for the time being. You then propose that "the resulting probability distribution of the inferred parameter is simply arising because of uncertainty, not inherent randomness in the parameter." Allow me to be even less rigorous in my mathematical terminology... I would say that "the resulting distribution" you speak of has some variance, and this final variance consists of two components: A) the variance originating from your prior probability distribution, and B) the variance originating from your sampling distribution. Your claim about variance "simply arising because of uncertainty, not inherent randomness in the parameter" can only apply component B of the final variance but not component A. Since there is some component of variance remaining in your final answer, even assuming your claim is true, your final answer must still be a random variable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.