Confusion about maximum likelihood estimation notation

Question

I'm trying to learn some machine learning theory, in particular maximum likelihood estimation.

Let me preface my question by saying that I'm very familiar with the Naive Bayes classifier, Bayes' Rule, and related notation. Below, I understand that $y$ is a class label and $X$ is a feature representation.

Now, my confusion stems from two different definitions of maximum likelihood estimation. Oddly, they are by the same author, Kevin P. Murphy.

Definition 1

Here's what he wrote in his 2012 book "Machine Learning: a Probabilistic Perspective", where he first describes $D$ (a training set of ($X, y$) pairs), and $H$, a hypothesis space, which I didn't understand. I was thinking that a hypothesis is a class label.

He then describes maximum likelihood estimation to be: $$ \hat{h}^{MLE} \triangleq \underset{h}{\mathrm{argmax}}\, P(D|h) $$

Definition 2

Now this is what he wrote in his upcoming 2021 book "Probabilistic Machine Learning: An Introduction". Here, MLE is applied to get the parameters $\theta$ that satisfy $\underset{\theta}{\mathrm{argmax}}\, P(D|\theta)$ .

Specific questions

1. Why are there two definitions of MLE, one to satisfy $\underset{h}{\mathrm{argmax}}\, P(D|h)$ and another to satisfy $\underset{\theta}{\mathrm{argmax}}\, P(D|\theta)$ ?

2. Is "hypothesis" a class label or a learnable parameter in these contexts?

3. In Bayes' Rule, we see the likelihood term as $P(X|y)$ with class label $y$. How does it relate to Definition 2, where we see the likelihood term $ P(D|\theta)$ with parameters $\theta$ instead of label $y$ ?

Answer 1

I'm trying to learn some machine learning theory, in particular maximum likelihood estimation.

At risk of being pedantic, but for the avoidance of doubt, I have found it useful to be clear on which field a set of tools comes from. In this case, whilst maximum likelihood estimation is ubiquitous in machine learning, there is general consensus that it originates in classical statistics, specifically, the work of Fisher and associates.

Why are there two definitions of MLE, one to satisfy $\text{argmax}_h P(\mathcal{D}|h)$ and another to satisfy $\text{argmax}_{\theta} P(\mathcal{D}| \theta)$ ?

These are equivalent formulations. That is, you can informally view a hypothesis class $\mathcal{H}$, a function class $\mathcal{F}$, and a parameter space $\Theta$ as the same thing. Where $h \in \mathcal{H}$, $f \in \mathcal{F}$ and $\theta \in \Theta$.

Which of these is used is largely a sub-field specific notational convention. So in the statistical learning theory literature, you will see hypothesis class and function class used interchangeably. Whereas in say the statistics literature, you will see the usage of a parameter space $\Theta$ used much more prevalently.

That they mean the same thing is often notationally represented as $h_{\theta}$ and $f_{\theta}$, meaning that we can index every function $f$ or hypothesis $h$ in the classes using a value of the parameter $\theta$ in the parameter space $\Theta$.

Is "hypothesis" a class label or a learnable parameter in these contexts?

Bearing the previous paragraph in mind, a single hypothesis $h$ corresponds to a particular value $\theta = \theta_0$. A set of hypotheses $h_{\theta}$ is represented by allowing $\theta$ to vary in $\Theta$.

In Bayes' Rule, we see the likelihood term as $P(X|y)$ with class label $y$. How does it relate to Definition 2, where we see the likelihood term $P(\mathcal{D}|\theta)$ with parameters $\theta$ instead of label $y$?

The usage of $P(\mathcal{D} | \theta)$ such as in defintion 2 refers to a (conditional) likelihood function, in a frequentist statistical sense. Given that in this setting a parameter $\theta$ is a fixed unknown number to be estimated, I find this notation somewhat confusing in the suggestion that we can condition on that which is not a random variable. Therefore you will also often see this denoted as $L(\theta)$ or $p(\mathcal{D} ; \theta)$.

There is also a likelihood function $P(\mathcal{D} | \theta)$ in a Bayesian statistical sense. Where the difference from above is that one now treats $\theta$ as a random variable on which we can condition.

The "likelihood" you have referred to as $p(X | y)$ involving conditioning on a class label $y$ is similar to the Bayesian likelihood, but is also distinct in that it is particular to naive Bayes classifiers. As indicated in your extract, it is best to refer to this as a class-conditional likelihood to avoid confusion.

Answer 2

Extending my comment:

1. In definition 2, the hypothesis here is that the vector of parameters takes on a particular value. In this regard, the notions are equivalent.
2. A hypothesis is not a class label. It could be a claim about a particular setting of model parameters. It could also be a claim about a hyperparameter, or the choice of model. In this context, for simplicity, just think of it according to point (1).
3. On the slide you screenshotted (please format math with LaTeX for blind and visually impaired users who rely on screen readers), those distributions are implicitly conditioned on $\boldsymbol{\theta}$. They’re not irrelevant, just unwritten. (Your course’s textbook will mention that somewhere.) The left-hand side of the expression on the slide is really $$ p(y \mid X, \boldsymbol{\theta})\text{.}$$ From this and Bayes’s rule, the relationship to the formula from Murphy’s book should be straightforward.

Note that you could also formulate different hypotheses than the ones I defined above. You might hypothesize about the form of the model, not only the parameter values. (For example, is the coin that generated this sequence of flips truly fair ($h_0$), or does it have an unknown parameter $p$ ($h_1$)?) In this case, your marginal likelihood requires integrating out the nuisance parameters.