I've got a problem, which I feel is related to this post. However I cannot really grasp the similarity and adopt it.
I am trying to deploy a model for object detection from video. The model outputs (from each frame) a probability of an object belonging to $n$ classes. For the final prediction I use the class with maximum probability. Usually the system works fine, however at some frames it outputs a different class than "expected".
For example: I have a frame at time $t_{-1}$ showing a back of a car (like a dash cam) - the output is a car with 0.9 probability and a cow with 0.08 probability (the rest of the classes are 0.02 in total). The output of the next frame at $t_0$ (which visually is no different from the one at $t_{-1}$) predicts a car with 0.4 probability, and a cow with 0.5 probability. The later frame at $t_{+1}$ is back at "normal" - similar to the frame at $t_{-1}$.
What I want to do is to use the probability of "neighboring" frames ($t_{-1}$, $t_{+1}$) to change the probability distribution of the middle frame.
For now I thought of three approaches:
1. I will average all three distributions and normalize it afterward.
2. I will multiply all three distributions classwise and normalize it afterwards.
3. I will use a Bayes rule to update the middle frame distribution according to the evidence obtained from neighboring frames.
Problems:
1. This solution basically says that I can tread the middle frame as "corrupted" and discard it, preserving only an average of the neighboring frames. (Well, I don't really want to do that.)
2. This solution seems better, because the neighboring frames "model" the prior distribution. However it still doesn't feel quite right.
3. This solution is quite easy to understand, however I have no idea how to formulate it in a mathematical form. What I get from the Bayes rule is:
$$
P{(t_0 | t_{-1} , t_{+1})} = \frac{P{(t_{-1} , t_{+1} | t_0)}P(t_0)}{P(t_{-1}, t_{+1})}
$$
where $P{(t_0 | t_{-1} , t_{+1})}$ is the posterior probability given evidence (which makes sens), $P(t_0)$ is the prior probability of the middle frame (an output of the model), $P{(t_{-1} , t_{+1})}$ is somehow a combination of the output of the model for neighboring frames (I don't know if it is right though, it should be a normalization factor and I don't know if they are the same things). I have the most severe issue with the factor $P{(t_{-1} , t_{+1} | t_0)}$ - I understand it as the probability of distribution of neighboring frames given the probability distribution of the middle frame. But I don't want the data to flow this way. I want to tread the neighboring frames as granted whether the middle frame is valid or not.
Any help is highly appreciated.
Edit:
I oversimplified the problem in order to explain it in a clear way. I am not looking for explanation why my model behaves like that (because in reality the number of classes > 10, and therefore a small change in distribution (attributed to noise etc) can shift the maximum to a different class, and I want to use the neighboring distribution to shift it back, UNLESS the model strongly "believes" that the distribution is correct).