I have two multivariate probability distributions, empirically observed. Here's some artificial data in two dimensions (with each value representing the number of events that occurred in that grid point/square):

N <- 20
p_obs <- matrix(sample.int(100, 100),N,N)
q_obs <- matrix(sample.int(100, 100),N,N)

The KL functions I've seen don't seem to care about the location of the grid points. For example, I can just turn the two matrices into vectors (losing the 2D structure), and compute KL like this:


#Make into vectors (losing grid position information)
p_vec <- as.double(p_obs)
q_vec <- as.double(q_obs)
p_vec <- p_vec / sum(p_vec)
q_vec <- q_vec / sum(q_vec)

x <- rbind(p_vec, q_vec)

Is this fine, or does KL need to know the structure of the grid?

(This is a 2D example, but I'm also interested in >2 dimensional cases)

EDIT: I know the p and q elements in the vectors must align with one another, but is that the only condition that’s required? (The loss of 2D structure is fine?)

  • $\begingroup$ The 2d structure is just a matter of how you store the data. In the end, you need to loop through all the entries, do the element-wise calculations, and sum them. It doesn't matter if you store the values in a vector, matrix, or multi-dimensional array. In R in fact it would be the same calculation in each case sum(p * log(p/q)), no matter of their shape. $\endgroup$
    – Tim
    May 17, 2021 at 19:05

1 Answer 1


KL divergence is

$$ D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right) $$

so to calculate it, you need to evaluate two probability distributions $P$ and $Q$ over the same values $x$. If you have the probabilities already calculated in a vector or a matrix, then yes, they need to be aligned.

  • $\begingroup$ Thanks, I know the elements must align inside the vector, but I’m wondering about the loss of 2D structure. Is it fine to put the vector elements in any order at all, as long as the values align? $\endgroup$
    – Mich55
    May 17, 2021 at 12:56
  • $\begingroup$ @Mich55 it is an implementational detail. You can store the values however you want as long as you can do the calculations properly, i.e. multiple, divide, and sum the values in a way that is consistent with the definition of KL. $\endgroup$
    – Tim
    May 17, 2021 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.