How does fitting a generative model ($P(X,Y)$) allow one to generate examples $(X,Y)$? For example, suppose I have a database of images of cats ($C$) and dogs ($D$). My database of labeled images consists of $(X,Y)$ where $X$ is a pixelated image and $Y \in \{C,D\}$. If I somehow fit a generative model to get the distribution $P(X,Y)$, how does that allow me to generate new examples $(X,Y)$ from the distribution?
For a simple case, it makes sense. I understand that knowing the distribution of dice rolls allows me to generate a "new" dice roll by picking one of the numbers $\{1,2,3,4,5,6\}$ with probability 1/6 for each. But how, in practice, does one apply a similar approach for generating an (image, label) pair $(X,Y)$ from the distribution $P(X,Y)$?
 A: In your example, you have two data generating processes.  The image generating process for cats, and the image generating process for dogs:
$P(X | Y=C)$ and $P(X | Y=D)$.  So, your generative model would have two components, one for cats, one for dogs.  If you want the probability of dogs to equal the probability of cats, $P(Y=C) = P(Y=D)$.
Given the class $Y$, you "draw" from the appropriate model.  To tie this to your simple example, if you had a fair die and a loaded die, you would pick which class to simulate, loaded or fair, then use that distribution, P(X | Y=loaded) or P(X| Y=fair).
When generating images, larger scale patterns across the pixels come into play. A good reference is

Mumford, David, and Agnès Desolneux. Pattern theory: the stochastic analysis of real-world signals. CRC Press, 2010.

However, the process is the same, pick your class, and draw a sample from that class.
$P(X,Y)$ is the joint distribution of $X$ and $Y$.  Once class probabilities $P(Y)$ are specified, it is computed as:
$$P(X,Y) = P(X | Y) P(Y)$$
as described in @DemetriPananos earlier comment.
$P(X | Y)$ are the generative model components: specify $Y=y$, draw $X$ from the $P(X | Y=y)$ model.
