KL divergence is defined between two distributions, period. If this is marginal or joint distributions is immaterial. You want them to have the same support. So you do the same as in single dimension. Asker in a comment says
Thanks for your useful answer. In the KL divergence, we must calculate
$\log p/q$ for probability distributions $p$ and $q$; how can this
calculation be done for joint distributions? (Sorry if this is a basic
$p$, $q$ are density functions, so they have values that are non-negative real numbers. So the quotient $p/q$ (assuming it is defined where needed, which it will be when the support of $p$ is included in the support of $q$) is a non-negative real number.
That conclusion does not at all depend on how many arguments the density functions $p, q$ have (they will normally have the same number of arguments). So the calculation of KL divergence is in principle the same for marginal and joint distributions, although the multivariate case might in practice be more involved.
For some examples see