Complex systems encompasses a very broad field so these are only pointers and not definite criteria. Complex systems lend themselves especially well if you are considering changes over time and non-linear relationships between three or more variables. Additionally, if your research questions or model of the phenomena you are studying require the consideration of complexity, such as: non-linearity, synchronization, chaos, phase transitions, and emergence you may want to consider it. Complex system can include a stochastic term, but do not always, if they do not they are termed deterministic.
a. The source of uncertainty in complex systems is usually different from the stochastic nature of uncertainty in statistical models. Most deterministic complex systems have uncertainty although they do not have an stochastic term, this is due to them being very sensitive to initial conditions. This is because many dynamics described by non-linear functions exhibit chaotic behavior.
b. Oftentimes the research question when investigating complex systems is different form the research question when using traditional statistical modeling (which usually deals with hypothesis testing and parameter estimation). This is because complex systems usually exhibit behaviors that are non-intuitive and depend on the interaction of many variables in unexpected ways. Thus the research question can be for example: how do relatively simple relationships between a system's elements explain a systems complex dynamics. This is sometimes termed emergence. Traditional statistical modeling does not account for this.
c. In the social sciences agent-based modeling, stochastic differential equations, and simulations based on social networks are often employed when examining such things as intractable conflicts and various aspects of individual social interactions (group formation, opinion polarization etc.). This kind of approach usually lends itself more easily to interactions on the individual level, so it traditionally probably used more in social psychology, whereas the group level usually is more easily described by adding a stochastic element, thus sociology traditionally relies more heavily on traditional statistical modeling (now I believe both fields rely on both).
SEMs and SEM-based methods do have a place in complex systems, but not a central one. SEM models do not allow for complexity. In the context of non-linearity SEM's can be thought of as linear approximations to well-behaved non-linear functions. Thus I guess you can use SEMs to estimate some parameters that will be later used in simulations or other complex systems analyses.
SEM's assume linear relationships between variables in the parameters, so effectively you can represent and estimate these using linear algebra (matrix and products of vectors, and matrixes). In such systems a small perturbation to the exogenous variables will cause a small difference in the endogenous variables. It does not exhibit complexity: the system is inherently linear (because it is made up of linear combinations of linear functions). For the social scientist the linearity in parameters assumptions of SEM can be too restrictive, especially when dealing with dynamics. Furthermore to explain a "complex" phenomena using SEM you need to have a large number of variables and parameters in your SEM model, complexity lets you describe "complex" phenomena using relatively few parameters.
3-4. I have glanced over Bar-Yam (1997) a couple years ago so I dont' remember too well. It seemed to be more theoretical with examples used to illustrate the techniques and concepts, I understand his newer book is more application driven, but maybe someone who actually read the books can compare these better. Chapters of the first book seem to be available online on the New England Complex Systems institute website.
Some terms you could search for include: dynamical systems, agent-based modeling, emergence, simulations in social sciences, and synchronization. Additionally I enjoyed book Sync by Stephan Strogatz, although it is more of an interesting read than a textbook, and I'm not sure it can be exactly called complex systems science, although it does encompass many interesting topics from it and is not math intensive.