Likelihood term in Bayesian inferencing versus the general definition

Question

In general we say that the likelihood function is defined as some $L(\theta|x)$, so that it is a function over some parameters: $\theta$ given some data: $x$. That is, $\theta$ is free to vary whilst $x$, our data, is fixed. This makes sense.

In Bayesian inferencing we claim that the term $P(x|\theta)$ is the likelihood term. However what I don't understand is that this term/function in Bayes' rule has fixed the $\theta$ variable, and so it is no longer the varying term. Instead we have a function varying over our data: $x$. And in reality often our data is given and fixed for us so that $x$ is fixed also. So technically the term $P(x|\theta)$ has both $\theta$ and $x$ non-varying.

I therefore am not sure how to interpret the $P(x|\theta)$ term properly. It is not varying over $\theta$ (unless we include the prior $P(\theta)$, but I feel like that would be cheating), and in addition to fixing $\theta$, $x$ is fixed also, so is it correct to even call $P(x|\theta)$ the likelihood term under these realities?

Answer 1

You want to have the probability over the parameters $\theta$ of your model given an outcome x.

Intuition : before having collected the outcome, you had in mind a probability over $\theta$. When you learn x, you must modify that probability by increasing the probability of parameters that better "explain" the outcome (i.e make it more likely, hence the name likelihood).

Calculus : You use the Bayes Rule (as you mentioned). What you want is $P(\theta/x)=P(x/\theta).P(\theta)/P(x)=L(\theta/x).P(\theta)/P(x)$

In that expression :

$P(x)$ does not depend on $\theta$.

$P(\theta)$ is your prior i.e. the probability over $\theta$ you had before having learned the outcome x. (Not always easy to assess).

$L(\theta/x)=P(x/\theta)$ is the likelihood term.

In the likelihood term, $\theta$ is NOT FIXED. I think that is what was not clear for you. The fact that $\theta$ is the condition in the probability does not mean that we should see it as a fixed value. The likelihood is like an award that is given to the parameters that better "explain" the outcome than others. Combined with the prior probabilities of the parameters they give the posterior probabilities.

Answer 2

If you are actually talking about a likelihood function, then the data vector $\mathbf{x}$ is considered fixed (i.e., not an argument of the function) and the parameter vector $\theta$ is a variable function argument. Usually you would denote the function by $L_\mathbf{x}: \Theta \rightarrow \mathbb{R}_+$ for a given data vector $\mathbf{x}$ so that $L_\mathbf{x}(\theta)$ gives the output of the function for the argument $\theta$. Bayes' rule can then be expressed in as:

$$\pi(\theta| \mathbf{x}) \propto L_\mathbf{x}(\theta) \cdot \pi(\theta).$$

In your question you denote the likelihood function with the non-standard notation $P(x|\theta)$, which is more usually used to denote the sampling density (and even then it is non-standard to capitalise the $P$). Since you don't specify a source for this notation, it is unclear if you saw this in a book somewhere, or if it is your own notation. In any case, if you intend for this to denote the likelihood function, as opposed to the sampling density, then you should interpret it as being a function of $\theta$ for a fixed set of data.