Does it make sense to use the KL-divergence between joint distributions of synthetic and real data, as a evaluation metric?

Question

The KL-divergence is defined as:

$D_{KL}(p(x_1)∥q(x_1))=\sum p(x_1)\, \log \Big( \dfrac{p(x_1)}{q(x_1)} \Big)$

I consider the Kullback-Leibler (KL) divergence as a performance metric for data synthesis.

Several studies used the KL divergence as a performance metric by computing over a pair of the real and synthetic marginal probability distribution for a given variable.

However, the joint distributions of variables in the synthetic and real data are also important for data synthesis.

If the real and synthetic data have variables of $x_1$ and $x_2$, does it makes sense to measure the KL-divergence between $P_{synthetic}(x_1,x_2)$ and $P_{real}(x_1,x_2)$ to evalaute the similarity between two datasets?

Thank you very much for your help!

P.S. Data synthesis is synthesizing the data using a Generative model (e.g., Variational autoencoder (VAE), Generative adversarial networks (GAN)), by estimating the joint probability distribution of the real data. Using the trained generative model, we can generate(synthesize or simulate) the synthetic data mimicking the real data. In the case of GAN, model training is conducted until the classification model in GAN cannot differentiate between the real and synthetic data. Therefore, I want to evaluate the similarity of joint distribution between synthetic and real data.

Answer 1

KL-divergence quantifies difference between two distributions, so it makes sense to use it to compare how realistic your synthetic data are. However, there are some practical considerations to take into account:

1. Unless the joint distributions that you consider are discrete, computing KL-divergence on empirical distributions will require some kind of density estimation and numerical integration. Binning the values and comparing histograms is a possible way to go. In this case, however, your results depend on the selected parameters. You should evaluate how stable your results are w.r.t. different binning.

2. To get a reasonable estimate of the KL-divergence, the number of samples needed grows exponentially with the number of variables that you consider. Evaluating bi-variate joint distributions might still be easy, but to get a good estimate of the divergence for a 10-variate distribution will require impractically large amount of samples.

3. KL-divergence is not symmetric. $KL[P_{real}\Vert P_{syn}] \neq KL[P_{syn}\Vert P_{real}]$