Each iteration of the EM algorithm will lead to a (weak) increase in likelihood. I however get tiny decreases of the (estimated) likelihood just before the parameter estimates converge. This is the first time that I use the EM algorithm, so I would be interested if this phenomeon can appear with a correctly specified and coded method or not and, if yes, if it is a common phenomenon.
Could it just be that the algorithm keeps searching in a flat area of the likelihood around the maximum, which leads to numerical decreases in the likelihood approximation? What are further diagnostics that I could check?
To get a picture, a typical sequence of changes in expected likelihood per observation that I get in simulated data looks something like this:
3.7311e-06 1.706e-06 1.4912e-05 9.1953e-05 0.00047164 0.0019663 0.0074215 0.021332 0.035066 0.046388 0.057377 0.062306 0.057089 0.040042 0.019623 0.0064888 0.0018022 0.00049184 0.00013265 3.4079e-05 7.7135e-06 1.1234e-06 -2.5607e-07 -3.8119e-07 -2.7317e-07 -1.6555e-07 -9.3877e-08 -5.1608e-08 -2.7939e-08 -1.501e-08 -8.0357e-09 -4.2965e-09 -2.2974e-09 -1.2294e-09 -6.5877e-10 -3.5356e-10 -1.9011e-10 -1.0242e-10 -5.5285e-11 -2.9929e-11 -1.6223e-11 -8.828e-12 -4.7957e-12 -2.6277e-12 -1.4402e-12 theta converged -- exiting
This question is a follow-up to that question.