On the relation between conditional distribution and likelihood

Question

Suppose we are modelling our data using a r.v. $X$ with pdf $f_X$ that depends on the parameter $\theta$. Further, we assume the parameter is itself a random variable, with a prior pdf of $f_{\theta^{old}}$

Let the joint distribution of the two be denoted: $f_{X,\theta^{old}}$. And, let's denote the conditional distribution as $f_{X|\theta^{old}}$. Let's denote the pdf of the posterior distribution as $f_{\theta^{new}}$.

Suppose we observe $X=x_0$.

We are told that:

$$f_{\theta^{new}}(\cdot) \propto \underbrace{L(\theta^{old} = \cdot; x_0)}_{likelihood}f_{\theta^{old}}(\cdot) \quad \ldots(\ast)$$

Now, using Bayes' theorem, we have:

$$f_{\theta^{new}}(\cdot) := f_{\theta^{old}|X=x_0}(\cdot) = \frac{\overbrace{f_{X|\theta^{old}}(x_0,\cdot)}^{conditional}f_{\theta^{old}}(\cdot)}{\int_{t}f_{X,\theta^{old}}(x_0, t) dt}$$

Which can also be written as:

$$f_{\theta^{new}}(\cdot) := \frac{1}{\int_{t}f_{X,\theta^{old}}(x_0, t) dt} \left(\frac{\overbrace{f_{X,\theta^{old}}(x_0, \cdot)}^{joint}}{\int_{x} f_{X,\theta^{old}}(x, \cdot)dx}\right)f_{\theta^{old}}(\cdot)$$

Now, in many textbooks, likelihood of a r.v. $X$ given an observation is taken to be the function got by treating the pdf as having variable parameters but fixed argument. If we consider a joint distribution over arguments and parameters like above, the likelihood, given observation $x_0$, appears to be this function:

$$ t \mapsto f_{X, \theta}(x_0, t)$$

So, in this case the constant of propotionality of $\ast$ is $=\int_{t}f_{X,\theta^{old}}(x_0, t) dt \int_{x} f_{X,\theta^{old}}(x, \cdot)dx$

But, in other places, the likelihood is taken to be the conditional probability of $X$ wrt $\theta$ with one argument fixed, viz:

$$ t \mapsto f_{X|\theta}(x_0, t)$$

In which case the constant of proportionality is simply the marginal distribution of $X$ evaluated at $x_0$

I'll be grateful if someone would clarify this situation.

Added: This problem doesn't arise in a frequentist context. There the likelihood function is unambiguous since it makes no sense to take a joint distribution of data and parameter or conditional distribution of the one over the other.

Answer 1

The point is moot as the likelihood is never defined as a joint distribution over observables and parameters.

...in many textbooks, likelihood of a r.v. $X$ given an observation is taken to be the function got by treating the pdf as having variable parameters but fixed argument.

The traditional definition of the likelihood function is the density of the sample $X$, at the observed value of the sample $x$, taken as a function of the unknown parameter when this parameter $\theta$ varies over the range of the parameter space.

[excerpt from Fisher (1922)]

If we consider a joint distribution over arguments and parameters like above, the likelihood, given observation $x_0$, appears to be this function: $$t↦f_{X,θ}(x_0,t)$$ So, in this case the constant of proportionality of (∗) is $$∫f_{X,θ^\text{old}}(x_0,t)\text{d}t∫f_{X,θ^\text{old}}(x,⋅)\text{d}x$$

This joint distribution $f_{X,\theta}$ is only meaningful in either a Bayesian or a fiducial setting, hence does not correspond to the meaning of a likelihood. (In the case of random effects and latent variables, the likelihood is the integral of a joint over the random effects or latent variables.) The function$$t↦f_{X,θ}(x_0,t)$$is then proportional to the posterior of $\theta$ given $x_0$ with normalising constant $$\int f_{X,θ}(x_0,t)\text{d}t$$but one cannot use it as a likelihood since (*) involves the multiplication of $L(\theta^{old} = \cdot; x_0)$ by the prior $f_{\theta^{old}}$.

Answer 2

I would offer this as a comment, but apparently I don't have the rep yet to comment.

I think that parameterizing time (outside of $\theta$) in the above makes things more complicated than necessary to understand - and perhaps the root of your confusion. Further, I don't think it is necessary to introduce notions of old and new data (and hence $\theta^{old}$ and $\theta^{new}$) as this seems to be a question about definitions of likelihood and marginal likelihood. I will simplify notation a bit first to make it more intuitive.

If your question is about updating parameters in light of new data, then my answer is superfluous, but I don't think it is.

Would I be correct in asserting that your question is essentially:

What is the difference between likelihood of parameters $\theta$ given data, and the likelihood likelihood of parameters $\theta,t$ given data?

Consider some data $D$ and a model $m$, where $m$ is parameterised by $\theta$. The probability of the data or likelihood of parameters is $P(D\vert\theta,m)$, which is equivalent to your $t \mapsto f_{X|\theta}(x_0, t)$. However, in your formulation, $t$ is also a parameter that you must marginalise out as well when defining the marginal likelihood. For now, consider $t\in\theta$ and write the marginal likelihood of the data as

$P(D\vert m)=\int P(D\vert \theta ,m) P\theta\vert m)d\theta$.

where it is easy to see that adding another parameter (time, $t$) would require a double integral, such as yours above.

However, I think I might have misunderstood your question - hence my wish to simply comment.