How is $P(D;\theta) = P(D|\theta)$?

Question

I started reading about maximum likelihood estimator and Bayesian statistics recently. I understand that given a statistical model $(X, (P_\theta))$ where, $\theta$ belongs to a large parameter space $\Theta$, the KL divergence between the $P_\theta$ and $P_\theta*$ ($\theta^*$ being the true parameter we would like to find) is minimised for the $\theta$ that maximises $\prod_{i=1}^{n}p_\theta(X_i)$. Assuming the events are independent and identically distributed, this amounts to maximising the the joint probability $P[X_1=x_1, X_2=x_2, ...,X_n=x_n].$ (the independence assumption allows to equate this to the product of the individual elements)

The Bayesian approach, accounts for the prior belief in the distribution of $\theta$, $P(\theta)$ and maximises $P(\theta|X)$, which by Bayes rule is equivalent to maximising, $P(X|\theta)P(\theta)/P(X)$. I understood things up to this part. After this, the $P(X|\theta)$ is called the "likelihood" and is replaced by $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$, which is just the product the individual probabilities of the X's in the distribution $P_\theta$. Does this mean that $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$ is actually $P_\theta[X_1=x_1, X_2=x_2, ...,X_n=x_n]$, i.e probabilities given $\theta$, or something like that ?

I'm not very good at probability and distributions, and my understanding is that the object $P(X|\theta)$ is called conditional probability, and the object $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$ (that equals $\prod_{i=1}^{n}p_\theta(X_i)$ by independence) is called the joint probability and they are very different things. I have seen authors use $P(X;\theta)$ for the joint probability in maximum likelihood in some cases. I'm confused why the joint probability and the conditional probability are considered to be equal ?

Answer 1

There are a couple of issues here:

1. In classical statistics all the distributions used are implicitly conditional on $\theta$, which is considered to be an "unknown constant". In Bayesian analysis there is no such thing as an unknown constant (anything unknown is treated as a random variable) and we instead use explicit conditioning statements for all probability statements.

2. This means that, in Bayesian analysis, the sampling density $P(X|\theta)$ is the object $P_\theta(X)$ that you referred to in the classical case. (The likelihood function is just the sampling density treated as a function of the parameter $\theta$ with $X=x$ taken to be fixed.) It also means that the density $P(X)$ in the Bayesian analysis is not conditional on $\theta$. It is the marginal density of the data, which is given by: $$P(X) = \int \limits_{\Theta} P(X|\theta) P(\theta) \ d \theta.$$ There are a few places in your question where you get a bit sloppy with conditioning statements, and you end up equivocating the conditional and marginal distributions of the data. That is not a big problem in classical statistics (since all probability statements are implicitly conditional on the parameter), but it will cause trouble for you in Bayesian analysis.

3. The notation $P(X ; \theta)$ is usually used only in classical statistics, and it is used to denote the same thing as $P_\theta(X)$ ---i.e., it is implicitly the conditional density of the data given the parameter. It would be unusual (and confusing) to use this notation for the joint density.

4. The Bayesian method whereby you maximise the posterior distribution with respect to the parameter is a point-estimation method called maximum a-posteriori (MAP) estimation. This is a point-estimation method that gives you a single point-estimate. You should bear in mind that Bayesians are usually concerned with also retaining the whole posterior density, since this contains more information than the MAP estimator.

Answer 2

I'll use a simplified notation in this answer. If you're doing classical statistics, $\theta$ is not a random variable. Hence, the notation $p(x;\theta)$ is describing a member of a family of probability functions or densities $\{p_\theta(x)\}_{\theta\in\Theta}$, in which $\Theta$ is the parameter space. In a Bayesian analysis, $\theta$ is a random variable, and $p(x\mid\theta)$ is a conditional probability function or density, which models your uncertainty about $x$ for each possible value of $\theta$. After you're done with your experiment, there is no longer uncertainty about $x$ (it becomes data/information you know about), and you look at $p(x\mid \theta)=L_x(\theta)$ as a function of $\theta$, for this "fixed" data $x$. This likelihood function $L_x(\theta)$ lives in the intersection between the classical and Bayesian styles of inference. In my opinion, the Bayesian way is better understood in terms of conditional independence. I suggest that you write down and explore the likelihood function for the Bernoulli model; graph it; think about it's meaning before and after the experiment is conducted. You mentioned that a Bayesian maximizes the posterior $\pi(\theta\mid x)$. That's not necessarily the case. There are other ways to summarize the posterior distribution. Essentially, the chosen summary depends on the introduction of a loss function. Check Robert's Bayesian Choice to learn all the gory details.