I have a model $p(x)$. How can adding latent variables $z$ help me? What are the main benefits of modelling $p(x, z)=p(x|z) p(z) $ instead of $p(x) $ alone? What would be some examples where modelling the latter would make my model better?
There are some elements to answer your question in Section 16.5 of the Deep Learning book by Ian Goodfellow and al.:
A good generative model needs to accurately capture the distribution over the observed or “visible” variables $v$ . Often the different elements of $v$ are highly dependent on each other. In the context of deep learning, the approach most commonly used to model these dependencies is to introduce several latent or “hidden” variables, $h$. The model can then capture dependencies between any pair of variables $v_i$ and $v_j$ indirectly, via direct dependencies between $v_i$ and $h$, and direct dependencies between $h$ and $v_j$.
The section also opposes the approach of adding latent variable to that of trying to model $p(v)$ without any latent variable:
A good model of v which did not contain any latent variables would need to have very large numbers of parents per node in a Bayesian network or very large cliques in a Markov network. Just representing these higher order interactions is costly. [...]
As an approach to discover such relevant (and computationaly tractable) interactions between the visible variables, the concept of structure learning is introduced. In general, modeling a fixed structure with latent variables avoids the need of structure learning between the visible variables. The book seems to imply that the former is easier than the latter. Indeed, we find later on this sentence:
Using simple parameter learning techniques we can learn a model with a fixed structure that imputes the right structure on the marginal $p( v )$.
Edit (thanks to carlo's comment): Going further in the analysis of structures with latent variables, we come accross the notion of interpretability. Jumping to Section 16.7, we can read:
When latent variables are used in the context of traditional graphical models, they are often designed with some specific semantics in mind—the topic of a document, the intelligence of a student, the disease causing a patient’s symptoms, etc. These models are often much more interpretable by human practitioners and often have more theoretical guarantees [...]
In many cases the data we observe depends on some hidden variables, that were not observed, or could not be observed. Knowing those variables would simplify our model, and in many cases we can get away from not knowing their values by assuming a latent variable model, that can "recover" the unobserved variables from the data.
Among popular examples of such models are finite mixture models, which assume that the data is clustered, while the cluster assignment is unknown and to be learned from the data. Those models can be used to learning distribution of the data, or more complicate cases like regression. In each case, the model learns to distinguish among several groups in the data, that share common characteristics, and fits the sub-models per each group, no matter that the group assignment was not known a priori. In plain English, instead of needing to build a complicated one-size-fitts-all model, you are building a model that consists of several, problem-specific, simpler models.
Another popular example is principle components analysis (see e.g. chapter 12 from Pattern Recognition and Machine Learning book by Bishop, 2006), or basically any other dimensionality reduction model, that are used to "compress" the observed data to smaller number of dimensions without much loss of information. Here the latent variables are the unobserved "features" of the data, that almost fully explain it. We are aiming at finding those features.
You can find very different example in my recent question, where we observed an aggregated data, while wanting to learn about the individual-level variability. As pointed in the answer, this can be thought as latent variable model, where we treat the predictions for the individuals as latent variables, that get aggregated, so that we can predict the aggregated responses to train the model. So contrary to previous examples where we used latent variables to find some higher-level features, here we use them to make lower-level, de-aggregated predictions. Again, here the individual-level values were not observed, so we replaced them with latent variables placeholders and made the model to predict them from the data.
Those are just few examples that illustrate latent variable models. You can find some more in the books by Bishop (2006), or Machine Learning: A Probabilistic Perspective by Kevin P. Murphy, who give many more examples and detailed explanations.
As a sidenote, it is worth mentioning, that those models can be in many cases hard to identify and often need some problem-specific computational tweaks and algorithms, so "guessing" the data that was not observed comes at some cost.