# Estimate core state transition probability matrix in partially observable Markov decision processes

I have a longitudinal data set of patients who have been monitored by a medical test over a year. The results of this test have false positive and false negative, so the system is partially observable. I want to estimate the transition probability matrix for the core state (the underlying state transitions) based on the observations in the data set.

I also have the specificity and sensitivity values for the medical test.

I have searched many things, including MSM package in R and EM algorithm, but none of them give me the transition probability matrix for core states. I would be very thankful if anyone has any relevant idea.

If I understand well, you observed the result of a medical test over time for each patient. I am assuming the result of the medical test is binary (positive-negative), so that your data consist of a $n\times t$ dataset db of $\{0,1\}$ values for $n$ patients observed over $t$ times. You assume a latent Markov process having 2 latent statuses (the "true" statuses or core statuses of the patients) and you want to estimate the latent transition probabilities. In statistics this kind of Hidden Markov model is often referred to as Latent Transition Analysis. An R package for LTA is LMest. So if db is your dataset:

library(LMest)
res <- est_lm_basic(db, yv=rep(1,n), k=2, start = 0, mod = 1)


mod = 1 indicates a time-homogeneous Latent Markov Chain
yv=rep(1,n) is the vector of frequencies of each observed $t$-sequence (here I put 1 for each observation without counting them)
k=2 is the number of latent statuses

As a result, res$Psi are the conditional probabilites, i.e., the probabilities of False Positive FP, True Negative TN, ecc.., while res$Pi are the latent transition probabilties you want.
Note that for the label switching problem (typical of finite mixture models), your latent statuses can be switched, that is, res$Psi can be : $$\left( \begin{array}{cc} TN & FN\\ FP & TP \end{array} \right) \text{ or } \left( \begin{array}{cc} FN & TN\\ TP & FP \end{array} \right)$$ but since you know the expected specificity and sensitivity of your test you should be able to distinguish them. I tried a simulation with$n=1000$and$t=15\$ and it works perfectly well.

FYI you can add individual level covariates (as gender or age) with est_lm_cov_latent.