HMM: Why are observations conditional on the latent state and not vice versa? The model of a HMM consists of a latent Markov chain with state $X$ and transition probabilities $P(X^t \mid X^{t-1})$, and observation variables $Y$ that depend on the current latent state via $P(Y^t \mid X^t)$.
This results in the following causal graphical model:

Question
What are reasons for why the arrows go from the latent state to the observed variable, and not the other way round?
I'm looking both for technical and philosophical reasons.

*

*An example of a technical reason could be: This way, the model has fewer parameters.

*Philosophical reasons could be based on the relationship between the causal structure of the model and the causal structure of reality. Shall the latent variable cause the observation, or shall they influence the latent state?

I'd also be very glad about pointers to literature that discusses this issue.
 A: Intuitively, the direction of the arrow makes sense because usually one interprets the observed part of the HMM as "symbols emitted by the hidden states".  
However, it's worth noticing that bayesian networks are not necessarily causal graphs.  Bayesian networks per se are simply one form of probabilistic  graphical models, which graphically encode conditional independencies, which allows you to factorize the joint distribution. 
So in this case the main goal of the bayesian network is to encode the fact that $Y_t$ is independent of $X_{t-1}$ given $X_t$. Or that $X_t$ is independent of $X_{t-2}$ given $X_{t-1}$ and so on. 
To make the point clear, it's also very common to represent a Hidden Markov Model with another form of graphical model --- Markov Random Fields, and in this case the edges are undirected, as shown below:

In sum, the main point of the graph in the case of HMM is to encode the conditional independencies, not usually associated with causal inference per se. Of course, if you want, you could interpret them causally if that makes sense on the context of your application.
