17
$\begingroup$

I know what a conjugate prior is. But I'm confused by the name itself. Why is it called "conjugate"? A complex conjugate $z^\ast$ has a reciprocal relationship with $z$, i.e., ${z^\ast}^\ast = z$. But there isn't such a reciprocal relationship between any two elements of the triad (prior, likelihood, posterior) or at least I'm not aware of it. So why "conjugate"? Is the term overloaded?

$\endgroup$

2 Answers 2

21
$\begingroup$

The Oxford English Dictionary defines "conjugate" as an adjective meaning "joined together, esp. in a pair, coupled; connected, related" and the statistical use is in that sense.

Many fields use it similarly:

  • chemistry: an acid that donates a proton to a base is a "conjugate acid"; whatever is left over is called a "conjugate base"
  • botany: "conjugate leaves" grow in pairs, especially when there's only one such pair,
  • optics: "conjugate foci" are the points where light is emitted and/or converges on opposite sides of a lens.
  • linguistics: conjugations are forms of the same root word.

A related word, conjugal, often refers to romantic/life partner in legal contexts.

Some of these senses do have a "reciprocal" implication, but it's often figurative, rather than strictly mathematical, and it doesn't seem like an essential part of the definition. It's not a huge stretch to imagine that a conjugate prior has a special and strong connection to its posterior.

Wikipedia credits Raiffa and Schlaifer for coining the term (annoyingly, it's not in the OED). Here's the first mention of it in their 1961 book Applied Statistical Decision Theory (worldcat,full text) which seems to be using the "related or joined" sense:

Screenshot of preface from Raiffa and Schlaifer

We show that whenever (1) any possible experimental outcome can be described by a sufficient statistic of fixed dimensionality (i.e., an $s$-tuple $(y_1, y_2, \ldots y_s)$ where $s$ does not depend on the "size" of the experiment), and 2) the likelihood of every outcome is given by a reasonably simple formula with $y_1, y_2, \ldots y_s$ as its arguments, we can obtain a very tractable family of "conjugate" prior distributions simply by interchanging the roles of variables and parameters in the algebraic expression for the sample likelihood, and the posterior distribution will be a member of the same family as the prior."

$\endgroup$
2
  • 1
    $\begingroup$ Very good point, this is also the earliest reference I am aware of. I would presume they used "conjugate" in a vague meaning of "associated" rather than more mathematically. $\endgroup$
    – Xi'an
    Commented Jul 25, 2020 at 14:46
  • $\begingroup$ I think that's actually the original meaning, with the "reciprocal"/ mathematical sense coming later as a special case. After all, one obvious way for things to be paired is for one to be the reciprocal of the other. $\endgroup$ Commented Jul 25, 2020 at 16:51
-2
$\begingroup$

I believe the origin is somehow related to the following concepts:

  • eigenvector: a vector $ \mathbf{x} $ is called an eigenvector of a matrix $ \mathbf{A} $ if $ \mathbf{A}\mathbf{x} $ = $k\mathbf{x}$ , meaning $ \mathbf{A}\mathbf{x} $ has the same form as $ \mathbf{x} $ (just different by a scaling factor $k$ called eigenvalue of $ \mathbf{A} $), hope you start to see this is the same logic with conjugate prior.

  • eigenfunction: see this analogy between conjugate prior and eigenfunction. The concept of eigenvector is extended to functions in Functional Analysis. Given a linear transformation $L$ (eg. a differential or integral operator), its eigenfunctions are functions $f$ such that $Lf$ is simply $kf$, ie. $f$ scaled by a scalar. The eigenfunctions are very useful in solving differential equations as they provide us with very convenient representation of their solutions. These are also related to Fourier transforms, where eigenfunctions of a Fourier transform are sine and cosine functions. In fact, it can be proved that any periodical function can be approximated as a linear combination of sine and cosine functions. Also, Fourier transform of a Gaussian function is another Gaussian function, again same logic with conjugated prior.

$\endgroup$
3
  • 3
    $\begingroup$ can you please elaborate what you mean by the usage of conjugated of a function in this context. $\endgroup$ Commented Jul 25, 2020 at 12:42
  • 4
    $\begingroup$ I frankly fail to see the connection between eigenfunctions and conjugate priors. $\endgroup$
    – Xi'an
    Commented Jul 25, 2020 at 14:50
  • $\begingroup$ Hi @Xi'an, just edited to clarify the connection between the two concepts. $\endgroup$
    – Victor Luu
    Commented Jul 25, 2020 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.