Compute likelihood of state given multiple observations?

Question

I am trying to use Bayes formula to compute the likelihood of a given state given a collection of independent but not sequenced observations - knowing the priors and knowing the probabilities of being in the state given each observation.

In other words for pre-defined states $S = \{S_1,\dots,S_m\}$ and possible observations $O=\{O_1,\dots,O_n\}$, the prior and conditional probabilities are already known:

$$ P(S_i) = \pi_i $$ $$P(S_i|O_j) = \rho_{ij}$$

(Note the second formula is not the "observation emission probability" but rather the state given observation probability.)

Suppose now I have collected a set of $k$ observations $\Theta=\{\theta_1,\dots, \theta_k\}$ and wish to compute for each state $i$:

$$P(S_i | \Theta) = P(S_i | \theta_1,\dots,\theta_k)$$

i.e. to work out the most likely state given the observations. I am having a brainfade as trying to use Bayes formula directly goes nowhere:

$$P(S_i | \theta_1,\dots,\theta_k) = \frac{P(\theta_1,\dots,\theta_k|S_i)P(S_i)}{\alpha}$$

I see the relevance of Hidden Markov Models (HMMs) in the mathematics but there is no natural sequence I'm dealing with here it is just a collection of unordered observations. What's the correct way to compute this?? (Pretend it's a HMM with a flat transition probability matrix??)

Suppose I accepted some independence assumptions and continued the derivation as follows:

$$ = \frac{P(\theta_1|S_i)\cdot \dots \cdot P(\theta_k|S_i)P(S_i)}{\alpha} $$

Then applied Bayes formula again to each term:

$$ = \frac{\frac{P(S_i|\theta_1)P(\theta_1)}{P(S_i)}\cdot \dots \cdot \frac{P(S_i|\theta_k)P(\theta_k)}{P(S_i)} P(S_i)}{\alpha} $$

Seems to be getting a bit silly and now I have to come up with $P(\theta_i)$ priors too (possible but surprised I need this) - what's going on and what's the correct way to compute this??

Answer 1

This is correct: \begin{align}\mathbb P(S=S_i|\theta_1,\ldots,\theta_k) \propto \frac{\mathbb P(S=S_i|\theta_1)p(\theta_1)}{\pi_i}\cdots \frac{\mathbb P(S=S_i|\theta_k)p(\theta_k)}{\pi_i} \pi_i\\ \propto \mathbb P(S=S_i|\theta_1)\cdots \mathbb P(S=S_i|\theta_k) \pi_i^{-k+1} \end{align} and the $p(\theta_j)$'s play no role there.

Answer 2

It's a little bit confusing as at the beginning we define a relationship between the probability of a state given a single observation but now we are looking at a set of observations. Do we know with certainty that all observations come from the same state? Assuming that we do then the solution by Xi'an is correct.

Reproducing it here with a few extra steps, without loss of generality let $k=2$:

\begin{equation} \mathit{P}(S_i|\theta_1, \theta_2) = \frac{\mathit{P}(\theta_1, \theta_2|S_i)\mathit{P}(S_i)}{\mathit{P}(\theta_1, \theta_2)} \hspace{0.5cm} \text{(Bayes Theorem)} \end{equation}

\begin{equation} = \frac{\mathit{P}(\theta_1|S_i)\mathit{P}(\theta_2|S_i)\mathit{P}(S_i)}{\mathit{P}(\theta_1)\mathit{P}(\theta_2)} \hspace{0.5cm} \text{(Independence Assumption)} \end{equation}

\begin{equation} = \frac{\mathit{P}(\theta_1|S_i)}{\mathit{P}(\theta_1)}\frac{\mathit{P}(\theta_2|S_i)}{\mathit{P}(\theta_2)}\mathit{P}(S_i) \hspace{0.5cm} \end{equation}

\begin{equation} = \frac{\mathit{P}(S_i|\theta_1)}{\mathit{P}(S_i)}\frac{\mathit{P}(S_i|\theta_2)}{\mathit{P}(S_i)}\mathit{P}(S_i) \hspace{0.5cm} \text{(Bayes Theorem)} \end{equation}

And so in general:

\begin{equation} \mathit{P}(S_i|\theta_1,...,\theta_n) =\frac{\mathit{P}(S_i|\theta_1)}{\mathit{P}(S_i)}...\frac{\mathit{P}(S_i|\theta_n)}{\mathit{P}(S_i)}\mathit{P}(S_i) = \mathit{P}(S_i|\theta_1)...\mathit{P}(S_i|\theta_n)\mathit{P}(S_i)^{-(n-1)} \end{equation}