Likelihood term in Bayesian inferencing versus the general definition

In general we say that the likelihood function is defined as some $L(\theta|x)$, so that it is a function over some parameters: $\theta$ given some data: $x$. That is, $\theta$ is free to vary whilst $x$, our data, is fixed. This makes sense.

In Bayesian inferencing we claim that the term $P(x|\theta)$ is the likelihood term. However what I don't understand is that this term/function in Bayes' rule has fixed the $\theta$ variable, and so it is no longer the varying term. Instead we have a function varying over our data: $x$. And in reality often our data is given and fixed for us so that $x$ is fixed also. So technically the term $P(x|\theta)$ has both $\theta$ and $x$ non-varying.

I therefore am not sure how to interpret the $P(x|\theta)$ term properly. It is not varying over $\theta$ (unless we include the prior $P(\theta)$, but I feel like that would be cheating), and in addition to fixing $\theta$, $x$ is fixed also, so is it correct to even call $P(x|\theta)$ the likelihood term under these realities?

• Likelihood is used in parameter estimation so $\theta$ is not fixed. I don't understand your sentence about Bayes' rule. Aug 9, 2018 at 9:02
• Bayes Rule: P(theta|x) = P(x|theta)P(theta)/P(x) where P(x|theta) is the likliehood function. However in P(x|theta), theta is fixed, and x is also fixed (data is given to us). So I dont see P(x|theta) as a term which varies over theta. In Bayes rule this variation over theta only comes in through the prior (I sample a theta from P(theta) then pass it to P(x|theta) for evaluation. But P(x|theta) by itself doesnt allow this..... or so I think. I would just like to find the flaw in my understanding. Aug 9, 2018 at 17:47
• In p(x|theta), theta is not fixed. Your observation : x is more or less likely to happen given the parameters of your probabilistic model. Aug 10, 2018 at 7:51
• Why do you consider setting prior as "cheating"? $\theta$ is a random variable, so it needs a distribution. Moreover, if you don't like priors, then whyare you interested in Bayesian approach?
– Tim
Aug 10, 2018 at 9:11
• The likelihood function is a function of both arguments $x$ and $\theta$ and has the property $\int L (x,\theta) dx=g (\theta)<\infty$ (or something like this). Saying that one is "fixed" and the other "varies" doesn't make sense to me Aug 13, 2018 at 9:32

You want to have the probability over the parameters $\theta$ of your model given an outcome x.

Intuition : before having collected the outcome, you had in mind a probability over $\theta$. When you learn x, you must modify that probability by increasing the probability of parameters that better "explain" the outcome (i.e make it more likely, hence the name likelihood).

Calculus : You use the Bayes Rule (as you mentioned). What you want is $P(\theta/x)=P(x/\theta).P(\theta)/P(x)=L(\theta/x).P(\theta)/P(x)$

In that expression :

$P(x)$ does not depend on $\theta$.

$P(\theta)$ is your prior i.e. the probability over $\theta$ you had before having learned the outcome x. (Not always easy to assess).

$L(\theta/x)=P(x/\theta)$ is the likelihood term.

In the likelihood term, $\theta$ is NOT FIXED. I think that is what was not clear for you. The fact that $\theta$ is the condition in the probability does not mean that we should see it as a fixed value. The likelihood is like an award that is given to the parameters that better "explain" the outcome than others. Combined with the prior probabilities of the parameters they give the posterior probabilities.

If you are actually talking about a likelihood function, then the data vector $\mathbf{x}$ is considered fixed (i.e., not an argument of the function) and the parameter vector $\theta$ is a variable function argument. Usually you would denote the function by $L_\mathbf{x}: \Theta \rightarrow \mathbb{R}_+$ for a given data vector $\mathbf{x}$ so that $L_\mathbf{x}(\theta)$ gives the output of the function for the argument $\theta$. Bayes' rule can then be expressed in as:

$$\pi(\theta| \mathbf{x}) \propto L_\mathbf{x}(\theta) \cdot \pi(\theta).$$

In your question you denote the likelihood function with the non-standard notation $P(x|\theta)$, which is more usually used to denote the sampling density (and even then it is non-standard to capitalise the $P$). Since you don't specify a source for this notation, it is unclear if you saw this in a book somewhere, or if it is your own notation. In any case, if you intend for this to denote the likelihood function, as opposed to the sampling density, then you should interpret it as being a function of $\theta$ for a fixed set of data.