How is $P(D;\theta) = P(D|\theta)$? I started reading about maximum likelihood estimator and Bayesian statistics recently. I understand that given a statistical model $(X, (P_\theta))$ where, $\theta$ belongs to a large parameter space $\Theta$, the KL divergence between the $P_\theta$ and $P_\theta*$ ($\theta^*$ being the true parameter we would like to find) is minimised for the $\theta$ that maximises $\prod_{i=1}^{n}p_\theta(X_i)$. Assuming the events are independent and identically  distributed, this amounts to maximising the the joint probability $P[X_1=x_1, X_2=x_2, ...,X_n=x_n].$ (the independence assumption allows to equate this to the product of the individual elements)
The Bayesian approach, accounts for the prior belief in the distribution of $\theta$, $P(\theta)$ and maximises $P(\theta|X)$, which by Bayes rule is equivalent to maximising, $P(X|\theta)P(\theta)/P(X)$. I understood things up to this part. After this, the $P(X|\theta)$ is called the "likelihood" and is replaced by $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$, which is just the product the individual probabilities of the X's in the distribution $P_\theta$. Does this mean that $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$ is actually $P_\theta[X_1=x_1, X_2=x_2, ...,X_n=x_n]$, i.e probabilities given $\theta$, or something like that ? 
I'm not very good at probability and distributions, and my understanding is that the object $P(X|\theta)$ is called conditional probability, and the object $P[X_1=x_1, X_2=x_2, ...,X_n=x_n]$ (that equals $\prod_{i=1}^{n}p_\theta(X_i)$ by independence) is called the joint probability and they are very different things. I have seen authors use $P(X;\theta)$ for the joint probability in maximum likelihood in some cases. I'm confused why the joint probability and the conditional probability are considered to be equal ?
 A: There are a couple of issues here:


*

*In classical statistics all the distributions used are implicitly conditional on $\theta$, which is considered to be an "unknown constant".  In Bayesian analysis there is no such thing as an unknown constant (anything unknown is treated as a random variable) and we instead use explicit conditioning statements for all probability statements.

*This means that, in Bayesian analysis, the sampling density $P(X|\theta)$ is the object $P_\theta(X)$ that you referred to in the classical case.  (The likelihood function is just the sampling density treated as a function of the parameter $\theta$ with $X=x$ taken to be fixed.)  It also means that the density $P(X)$ in the Bayesian analysis is not conditional on $\theta$.  It is the marginal density of the data, which is given by: $$P(X) = \int \limits_{\Theta} P(X|\theta) P(\theta) \ d \theta.$$ There are a few places in your question where you get a bit sloppy with conditioning statements, and you end up equivocating the conditional and marginal distributions of the data.  That is not a big problem in classical statistics (since all probability statements are implicitly conditional on the parameter), but it will cause trouble for you in Bayesian analysis.

*The notation $P(X ; \theta)$ is usually used only in classical statistics, and it is used to denote the same thing as $P_\theta(X)$ ---i.e., it is implicitly the conditional density of the data given the parameter.  It would be unusual (and confusing) to use this notation for the joint density.

*The Bayesian method whereby you maximise the posterior distribution with respect to the parameter is a point-estimation method called maximum a-posteriori (MAP) estimation.  This is a point-estimation method that gives you a single point-estimate.  You should bear in mind that Bayesians are usually concerned with also retaining the whole posterior density, since this contains more information than the MAP estimator.
A: I'll use a simplified notation in this answer. If you're doing classical statistics, $\theta$ is not a random variable. Hence, the notation $p(x;\theta)$ is describing a member of a family of probability functions or densities $\{p_\theta(x)\}_{\theta\in\Theta}$, in which $\Theta$ is the parameter space. In a Bayesian analysis, $\theta$ is a random variable, and $p(x\mid\theta)$ is a conditional probability function or density, which models your uncertainty about $x$ for each possible value of $\theta$. After you're done with your experiment, there is no longer uncertainty about $x$ (it becomes data/information you know about), and you look at $p(x\mid \theta)=L_x(\theta)$ as a function of $\theta$, for this "fixed" data $x$. This likelihood function $L_x(\theta)$ lives in the intersection between the classical and Bayesian styles of inference. In my opinion, the Bayesian way is better understood in terms of conditional independence. I suggest that you write down and explore the likelihood function for the Bernoulli model; graph it; think about it's meaning before and after the experiment is conducted. You mentioned that a Bayesian maximizes the posterior $\pi(\theta\mid x)$. That's not necessarily the case. There are other ways to summarize the posterior distribution. Essentially, the chosen summary depends on the introduction of a loss function. Check Robert's Bayesian Choice to learn all the gory details.
