1
$\begingroup$

Suppose we are modelling our data using a r.v. $X$ with pdf $f_X$ that depends on the parameter $\theta$. Further, we assume the parameter is itself a random variable, with a prior pdf of $f_{\theta^{old}}$

Let the joint distribution of the two be denoted: $f_{X,\theta^{old}}$. And, let's denote the conditional distribution as $f_{X|\theta^{old}}$. Let's denote the pdf of the posterior distribution as $f_{\theta^{new}}$.

Suppose we observe $X=x_0$.

We are told that:

$$f_{\theta^{new}}(\cdot) \propto \underbrace{L(\theta^{old} = \cdot; x_0)}_{likelihood}f_{\theta^{old}}(\cdot) \quad \ldots(\ast)$$

Now, using Bayes' theorem, we have:

$$f_{\theta^{new}}(\cdot) := f_{\theta^{old}|X=x_0}(\cdot) = \frac{\overbrace{f_{X|\theta^{old}}(x_0,\cdot)}^{conditional}f_{\theta^{old}}(\cdot)}{\int_{t}f_{X,\theta^{old}}(x_0, t) dt}$$

Which can also be written as:

$$f_{\theta^{new}}(\cdot) := \frac{1}{\int_{t}f_{X,\theta^{old}}(x_0, t) dt} \left(\frac{\overbrace{f_{X,\theta^{old}}(x_0, \cdot)}^{joint}}{\int_{x} f_{X,\theta^{old}}(x, \cdot)dx}\right)f_{\theta^{old}}(\cdot)$$

Now, in many textbooks, likelihood of a r.v. $X$ given an observation is taken to be the function got by treating the pdf as having variable parameters but fixed argument. If we consider a joint distribution over arguments and parameters like above, the likelihood, given observation $x_0$, appears to be this function:

$$ t \mapsto f_{X, \theta}(x_0, t)$$

So, in this case the constant of propotionality of $\ast$ is $=\int_{t}f_{X,\theta^{old}}(x_0, t) dt \int_{x} f_{X,\theta^{old}}(x, \cdot)dx$

But, in other places, the likelihood is taken to be the conditional probability of $X$ wrt $\theta$ with one argument fixed, viz:

$$ t \mapsto f_{X|\theta}(x_0, t)$$

In which case the constant of proportionality is simply the marginal distribution of $X$ evaluated at $x_0$

I'll be grateful if someone would clarify this situation.

Added: This problem doesn't arise in a frequentist context. There the likelihood function is unambiguous since it makes no sense to take a joint distribution of data and parameter or conditional distribution of the one over the other.

$\endgroup$
  • $\begingroup$ @Xi'an I didn't mean defined as a joint density but as a function with the form of the joint density with data=observation. For ex: DeGroot and Schervish. I think my confusion partly stems from how to parse the density of X, viz; P(X=x; \theta), whether as a conditional distribution of X at a fixed parameter value or as the marginalization of the joint dist. wrt parameter. Sorry if this is stupid. :( $\endgroup$ – sntx Feb 28 '17 at 14:29
  • 1
    $\begingroup$ The issue is that there is no joint distribution without introducing a prior distribution. The likelihood is the part carrying the information brought by the data in a Bayesian analysis. $\endgroup$ – Xi'an Feb 28 '17 at 14:46
  • $\begingroup$ See also: stats.stackexchange.com/questions/224037/… $\endgroup$ – Tim Feb 28 '17 at 14:49
  • $\begingroup$ @Xi'an So, the conditional distribution of X wrt parameter is what is foundational; whereas the joint distribution is then found by scaling it with the prior distribution of the parameter? $\endgroup$ – sntx Feb 28 '17 at 14:55
  • 1
    $\begingroup$ @Xi'an Right. Thank you very much for clearing this up. I find the modelling the parameter with a r.v. very natural since the observed sample is what is given and our uncertainty (like in Poincare's essay) is about the parameter. $\endgroup$ – sntx Feb 28 '17 at 15:09
1
$\begingroup$

The point is moot as the likelihood is never defined as a joint distribution over observables and parameters.

...in many textbooks, likelihood of a r.v. $X$ given an observation is taken to be the function got by treating the pdf as having variable parameters but fixed argument.

The traditional definition of the likelihood function is the density of the sample $X$, at the observed value of the sample $x$, taken as a function of the unknown parameter when this parameter $\theta$ varies over the range of the parameter space.

Fisher, R.A. (1922). "On the mathematical foundations of theoretical statistics". Philosophical Transactions of the Royal Society A. 222 (594–604): 309–368.

[excerpt from Fisher (1922)]

If we consider a joint distribution over arguments and parameters like above, the likelihood, given observation $x_0$, appears to be this function: $$t↦f_{X,θ}(x_0,t)$$ So, in this case the constant of proportionality of (∗) is $$∫f_{X,θ^\text{old}}(x_0,t)\text{d}t∫f_{X,θ^\text{old}}(x,⋅)\text{d}x$$

This joint distribution $f_{X,\theta}$ is only meaningful in either a Bayesian or a fiducial setting, hence does not correspond to the meaning of a likelihood. (In the case of random effects and latent variables, the likelihood is the integral of a joint over the random effects or latent variables.) The function$$t↦f_{X,θ}(x_0,t)$$is then proportional to the posterior of $\theta$ given $x_0$ with normalising constant $$\int f_{X,θ}(x_0,t)\text{d}t$$but one cannot use it as a likelihood since (*) involves the multiplication of $L(\theta^{old} = \cdot; x_0)$ by the prior $f_{\theta^{old}}$.

$\endgroup$
0
$\begingroup$

I would offer this as a comment, but apparently I don't have the rep yet to comment.

I think that parameterizing time (outside of $\theta$) in the above makes things more complicated than necessary to understand - and perhaps the root of your confusion. Further, I don't think it is necessary to introduce notions of old and new data (and hence $\theta^{old}$ and $\theta^{new}$) as this seems to be a question about definitions of likelihood and marginal likelihood. I will simplify notation a bit first to make it more intuitive.

If your question is about updating parameters in light of new data, then my answer is superfluous, but I don't think it is.

Would I be correct in asserting that your question is essentially:

What is the difference between likelihood of parameters $\theta$ given data, and the likelihood likelihood of parameters $\theta,t$ given data?

Consider some data $D$ and a model $m$, where $m$ is parameterised by $\theta$. The probability of the data or likelihood of parameters is $P(D\vert\theta,m)$, which is equivalent to your $t \mapsto f_{X|\theta}(x_0, t)$. However, in your formulation, $t$ is also a parameter that you must marginalise out as well when defining the marginal likelihood. For now, consider $t\in\theta$ and write the marginal likelihood of the data as

$P(D\vert m)=\int P(D\vert \theta ,m) P\theta\vert m)d\theta$.

where it is easy to see that adding another parameter (time, $t$) would require a double integral, such as yours above.

However, I think I might have misunderstood your question - hence my wish to simply comment.

$\endgroup$
  • $\begingroup$ Hi. There is no notion of 'time-indexing' in my question. Perhaps my use of 'old' and 'new' has mislead you. They are just an intuitive labelling for the two distributions of the parameter. $\endgroup$ – sntx Feb 28 '17 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.