Bollen and Pearl (2013) in "Handbook of Causal Analysis for Social Research" treat factor analysis models (like CFA) as part of SEM.
In the path diagram, the ovals or circles represent the latent variables. As stated above, these are variables that are part of our theory, but not in our data set. As in the previous path diagrams, the observed variables are in boxes, single-headed arrows stand for direct causal effects, and two-headed arrows (often curved) signify sources of associations between the connected variables, though the reasons for their associations are not specified in the model. It could be that they have direct causal influence on each other, that some third set of variables not part of the model influence both, or there could be some other unspecified mechanism (preferential selection) leading them to be associated. The model only says that they are associated and not why. [...]
To my mind, CFA is a SEM model where you don't take any position on why or how the latent variables are correlated.