I am currently debugging a hand-coded implementation of a hidden Markov model, and as part of this, am scrutinising whether I have appropriately specified the log-likelihood computation algebraically. In particular, I would like to check whether my interpretations of "log-likelihood" in this context are appropriate, and what general conventions there are for reporting "log-likelihood", if any.
Queries.
1. If I were to report the joint log-likelihood of the hidden Markov model using state vectors $\mathbf{y}$ and observation vectors $\mathbf{x}$ am I correct in understanding that there is neither the need for a forward algorithm nor a backward algorithm?
For clarity, I am taking the meaning of "forward" and "backward" algorithm to refer to a procedure for efficient exact inference in a tree-structured probabilistic graphical model, involving sum-product message passing/belief propagation. And my query amounts to whether the following computation requires sum-product message passing, or is rather, merely a task of "plugging-in" $N$ values $\mathbf{x}$ and $\mathbf{y}$.
$$l_1(\Theta) = \prod^N_{n=1} p(\mathbf{x}^{(n)}, \mathbf{y}^{(n)}; \Theta)$$
My current position on this is that there is no need for sum-product message passing, given that there is no marginalisation involved, but for debugging purposes, I am in need of absolute clarity on this.
2. As a follow-up, if I were to report the log-likelihood of the hidden Markov model using state vectors $\mathbf{y}$ and observation vectors $\mathbf{x}$, am I correct in understanding that reporting the following log-likelihood computation DOES require a forward algorithm/sum-product message passing algorithm?
$$l_2(\Theta) = \prod^N_{n=1} p(\mathbf{x}^{(n)}; \Theta)$$
My current position on this is that there is a need for sum-product message passing, given that marginalisations are required. And that the above computation involves the marignalisations $\sum_{y_1} \dots \sum_{y_{t_n}} p(\mathbf{x}, \mathbf{y}; \Theta)$, whereby the number of marginalisations corresponds to the number of components in the state vector $\mathbf{y} = (y_1, ..., y_{T_n})$
3. Are there general conventions governing whether the phrase "log-likelihood" in context of a hidden Markov model refers to $l_1(\Theta)$ or $l_2(\Theta)$ in the notation above, or is it assumed that it is the responsibility of the relevant author to clarify this?
Perhaps this might be viewed as unnecessary hair splitting, but I am trying to reconcile the potential ambiguities in the meanings above in context of classical parametric statistics.
Because in frequentist parametric statistics, we normally say that the log-likelihood function $l(\Theta)$ is a function of the parameters $\Theta$, evaluated at the observed values of the data say $X_1 = x_1, ..., X_N = x_n$. However in the HMM case, our data consists of two sets of random variables $\mathbf{x} = (X_1, ..., X_{T_n})$ and $\mathbf{y} = (Y_1, ..., Y_{T_n})$, and we can choose to report either $l_1(\Theta)$ or $l_2(\Theta)$.
Further context.
I have a hidden Markov model, consisting of an initial distribution $\boldsymbol{\pi} \in \mathbb{R}^M$, a state transition matrix $\mathbf{A} \in \mathbb{R}^{M \times M}$, and an emission matrix $\mathbf{B} \in \mathbb{R}^{M \times V}$. That these parameters $\Theta = \{\boldsymbol{\pi}, \mathbf{A}, \mathbf{B} \}$ are all matrices implies that the hidden Markov model has discrete states, observations; and also time is discrete.
My training data set consists of $N$ observation-state vector pairs $\{(\mathbf{x}^{(n)}, \mathbf{y}^{(n)}) \}^N_{n=1}$, where both observation vector $\mathbf{x} = (x_1, ..., x_{T_n})$ and state vector $\mathbf{y} = (y_1, ..., y_{T_n})$ both have $T_n$ components, corresponding to a variable length state or observation sequence. Where this variability in sequence length is represented through indexing of the instance $n$.
