Parameter estimation of state-space models with hidden variables I have a time-series analysis problem, that I am having trouble finding a suitable regression technique for.
I have a coupled linear three dimensional system
\begin{align*}
X_{t} & =\left(1+J\right)X_{t-1}+GU_{t},
\end{align*}
where
\begin{align*}
J & =\left(\begin{array}{ccc}
J_{11} & J_{12} & 0\\
J_{21} & J_{22} & 0\\
J_{31} & J_{32} & 0
\end{array}\right)
\end{align*}
is a $3\times3$ Matrix (the Jacobian stemming from linearization
for context) describing the deterministic evolution (note that the third variable doesn't couple back), $U_{t}$ is an
$M$ dimensional vector of mutually independent gaussian noises with
variance $\sigma^{2}$, and $G$ is an $M\times3$ Matrix resulting
in the additive noise being a superposition of the indiviudial noise
entries.
The key feature is that only two of the three variables are observed,
which I may represent in the typical state space model / Kalman filter
manner by introducing an observation eqauation
\begin{align*}
Y_{t} & =H\,X_{t}
\end{align*}
with
\begin{align}H=\left(\begin{array}{ccc}
1 & 0 & 0\\
0 & 1 & 0\\
\end{array}\right)
\end{align}and no measurement noise.
In this form I understand that I can use a Maximum likelihood parameter
estimation of state space models via the Kalman Filter to find $J,$$G$
and $\sigma^{2}.$ My question now is if this is the best approach,
what caveats of the method may be and if someone could point me to
an implementation (preferrably in python). In my research I also stumbled
on the the claim that the EM algorithm (Expectation Maximization)
is the one to do the job. What does it do better?
 A: EM is one possible algorithm to do ML estimation, see, e.g., here. But you can also simply optimize the likelihood directly using any other standard optimization algorithm. (Be aware that the problem is in general nonconvex, though. Many people just ignore this, but I actually found it important in practice to use some kind of global optimization strategy.) Analytical expressions for the likelihood and its derivatives are available, see, e.g., chapter 12 of Simo Särkkä's excellent book "Bayesian filtering and smoothing". I have had good success doing that as have many others, you can find my Matlab implementation on github.
If you want to use a standard python library, statsmodels seems to support ML parameter estimation in state-space models. If you're a bit more adventurous, you could also use something like pyro or pymc, but that is probably a larger endeavour if you haven't worked with a probabilistic programming package before.
For another, more optimization-oriented perspective and an accompanying python implementation, see "Fitting a Kalman smoother to data" by Barratt and Boyd (2020) [arxiv] and the accompanying python code.
