# Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The objective is to select a subset of individuals with size $n$ $(n\leq N)$ such that the probability distribution of the measurable quantity, denoted by $P(\hat{X})$ where $\hat{X}\in\mathbb{R}^{n\times M}$, becomes almost equal to $P(X)$, for any $P(X)$.

To conduct the above optimization task, one criterion that I am using is to measure the distance between $P(X)$ and $P(\hat{X})$. This can be achieved by using the Kullback-Leibler divergence measure which is given by $$D_{KL}(p\| q)=\sum^{T}_{i=1} p(x_i) \log{\left\{\frac{p(x_i)}{q(x_i)}\right\}}$$ where $x_i$ is an element of the discrete space $\mathbb{L}^{N}$ and the cardinality of $\mathbb{L}^{N}$ is $T$. In this situation, the observed distribution $p$ corresponds to $P(\hat{X})$ while the theoretical distribution $q$ corresponds to $P(X)$. $D_{KL}(p\| q)$ is then a measure of the information lost (or error) when q is used to approximate p. Also, note that when $q(x_i)$ is zero then $p(x_i)$ will be zero as well, since the distribution $p(x_i)$ derives from $q(x_i)$, that is, $\hat{X} \subseteq X$.

Now lets say that there is in total 100 individuals and that the subset size is 90. This means that 10 individuals have been omitted and lets assume that the Kullback-Leibler divergence measure gives for this situation something like 0.2357.

My question now relates to the quantification of the error that the measure gives. In that, what does 0.2357 actually means? Could I say something like by omitting 10 individuals then 23.57% of the dataset variance has been lost? If no, is there any measure capable of estimating how much variance gets lost (or left unaccounted) when the mentioned individuals are omitted? Could the mutual information measure be utilized?