27
$\begingroup$

What is the difference in meaning between the notation $P(z;d,w)$ and $P(z|d,w)$ which are commonly used in many books and papers?

$\endgroup$
  • 13
    $\begingroup$ f(x;θ) is the same as f(x|θ), simply meaning that θ is a fixed parameter and the function f is a function of x. f(x,Θ), OTOH, is an element of a family (set) of functions, where the elements are indexed by Θ. A subtle distinction, perhaps, but an important one, esp. when it comes time to estimate an unknown parameter θ on the basis of known data x; at that time, θ varies and x is fixed, resulting in the "likelihood function". Usage of "|" is more common among statisticians, ";" among mathematicians. $\endgroup$ – jbowman Jun 20 '12 at 19:20
  • $\begingroup$ Yes jbowman is correct. We sometimes call it the density of X given Θ. $\endgroup$ – Michael R. Chernick Jun 20 '12 at 19:51
  • $\begingroup$ @jbowman why not post that as an answer? My only question is - why would they use both, but I assume that it has something to do with the context (the "|" is used with "P" and the ";" with "$f$"). $\endgroup$ – Abe Jun 21 '12 at 15:11
  • $\begingroup$ Good thinking, Abe; that's probably it. $f$ is more generic, I suppose. $\endgroup$ – jbowman Jun 21 '12 at 15:13
12
$\begingroup$

I believe the origin of this is the likelihood paradigm (though I have not checked the actual historical correctness of the below, it is a reasonable way of understanding how iot came to be).

Let's say in a regression setting, you would have a distribution: p(Y | x, beta) Which means: the distribution of Y if you know (conditional on) the x and beta values.

If you want to estimate the betas, you want to maximize the likelihood: L(beta; y,x) = p(Y | x, beta) Essentially, you are now looking at the expression p(Y | x, beta) as a function of the beta's, but apart from that, there is no difference (for mathematical correct expressions that you can properly derive, this is a necessity --- although in practice noone bothers).

Then, in bayesian settings, the difference between parameters and other variables soon fades, so one started to you use both notations intermixedly.

So, in essence: there is no actual difference: they both indicate the conditional distribution of the thing on the left, conditional on the thing(s) on the right.

$\endgroup$
23
$\begingroup$

$f(x;\theta)$ is the density of the random variable $X$ at the point $x$, with $\theta$ being the parameter of the distribution. $f(x,\theta)$ is the joint density of $X$ and $\Theta$ at the point $(x,\theta)$ and only makes sense if $\Theta$ is a random variable. $f(x|\theta)$ is the conditional distribution of $X$ given $\Theta$, and again, only makes sense if $\Theta$ is a random variable. This will become much clearer when you get further into the book and look at Bayesian analysis.

$\endgroup$
  • $\begingroup$ Uhhhh... $f(x|\theta)$ is the conditional distribution of $x$ given $\theta$ makes perfect sense even if $\theta$ is not a random variable. It's pretty much standard notation in classical statistics, where $\theta$ is not a random variable. $\endgroup$ – jbowman Jun 21 '12 at 17:10
  • $\begingroup$ Uhhhh....if you interpret that to mean that P[Θ=θ]=1 (left Θ is a random variable, right θ is a constant) then I agree. Otherwise I do not...for what then would P[Θ=θ] mean in the denominator of the definition of conditional distribution? $\endgroup$ – PeterR Jun 21 '12 at 17:30
  • $\begingroup$ Denominator? I can write $x \sim f(x | \mu, \sigma)$ where $f$ is a Normal distribution without reference to Bayes' Rule. $\mu$ and $\sigma$ are fixed. Others do too, for example, ll.mit.edu/mission/communications/ist/publications/…. $\endgroup$ – jbowman Jun 21 '12 at 18:28
  • $\begingroup$ jbowman, so what is the definition of your f(x|μ,σ) as a conditional density when μ and σ are fixed numbers (i.e. not random variables)? $\endgroup$ – PeterR Jun 21 '12 at 18:41
  • 1
    $\begingroup$ The word "conditional", associated with the notation f(X|Y), is defined to be "conditional upon some random event occurring". If you are using it to mean something else, such as just "given", as in "f(x) given (specific values of) μ and σ", well then that is what the notation f(x;μ,σ) is for. Since the OP was asking about what the notation means, we should be precise about the notation in the answer. $\endgroup$ – PeterR Jun 21 '12 at 19:49
18
$\begingroup$

$f(x;\theta)$ is the same as $f(x|\theta)$, simply meaning that $\theta$ is a fixed parameter and the function $f$ is a function of $x$. $f(x,\Theta)$, OTOH, is an element of a family (or set) of functions, where the elements are indexed by $\Theta$. A subtle distinction, perhaps, but an important one, esp. when it comes time to estimate an unknown parameter $\theta$ on the basis of known data $x$; at that time, $\theta$ varies and $x$ is fixed, resulting in the "likelihood function". Usage of $\mid$ is more common among statisticians, while $;$ among mathematicians.

$\endgroup$
  • 1
    $\begingroup$ How is $f(x;θ)$ spoken verbally? Do you say " f of x given θ"? $\endgroup$ – stackoverflowuser2010 Oct 23 '14 at 22:09
  • $\begingroup$ @stackoverflowuser2010 - yes, exactly so. $\endgroup$ – jbowman Oct 27 '14 at 20:47
  • 2
    $\begingroup$ I found in some Coursera videos that Stanford professor Andrew Ng verbalizes the semicolon as "parameterized by." See: class.coursera.org/ml-005/lecture/34 . So the example would be spoken as "f of x parameterized by theta". $\endgroup$ – stackoverflowuser2010 Nov 10 '14 at 2:50
  • 5
    $\begingroup$ Saying "given" or "conditional" is very different (in general) from "parameterized." I'd hate if someone saw this and thought the two were equivalent. Saying "parameterized" is only appropriate when the quantity being conditioned on is a parameter indexing the pdf of the variable in the first term. For two variables (e.g., f(x;y)), using that term would be wrong. $\endgroup$ – ATJ Jun 17 '16 at 17:07
  • 2
    $\begingroup$ @MikeWilliamson - Sure, pick a notation where you know what everything means and stick with it! That way when you go back to something you did earlier, like 4 hours earlier in my experience, you don't have to figure out what you meant when you used that "|". I agree, it is annoying, but after a while you just observe the first use of the notation and remember it for the rest of the paper / book; the distinctions are not usually what's important, anyway. $\endgroup$ – jbowman Aug 18 '18 at 18:42
9
$\begingroup$

Although it hasn't always been this way, these days $P(z; d, w)$ is generally used when $d,w$ are not random variables (which isn't to say that they're known, necessarily). $P(z | d, w)$ indicates conditioning on values of $d,w$. Conditioning is an operation on random variables and as such using this notation when $d, w$ aren't random variables is confusing (and tragically common).

As @Nick Sabbe points out $p(y|X, \Theta)$ is a common notation for the sampling distribution of observed data $y$. Some frequentists will use this notation but insist that $\Theta$ isn't a random variable, which is an abuse IMO. But they have no monopoly there; I've seen Bayesians do it too, tacking fixed hyperparameters on at the end of the conditionals.

$\endgroup$
  • 2
    $\begingroup$ Re your 2nd paragraph, it's worth pointing out that in typical statistical situations (say, fitting a regression model), $X$ isn't considered a random variable either, but a set of known constants. $\endgroup$ – gung - Reinstate Monica Aug 25 '13 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.