# Multivariate KL Divergence - does data ordering matter?

I have two multivariate probability distributions, empirically observed. Here's some artificial data in two dimensions (with each value representing the number of events that occurred in that grid point/square):

N <- 20
p_obs <- matrix(sample.int(100, 100),N,N)
q_obs <- matrix(sample.int(100, 100),N,N)



The KL functions I've seen don't seem to care about the location of the grid points. For example, I can just turn the two matrices into vectors (losing the 2D structure), and compute KL like this:

library(philentropy)

#Make into vectors (losing grid position information)
p_vec <- as.double(p_obs)
q_vec <- as.double(q_obs)
#Normalise
p_vec <- p_vec / sum(p_vec)
q_vec <- q_vec / sum(q_vec)

x <- rbind(p_vec, q_vec)
KL(x)



Is this fine, or does KL need to know the structure of the grid?

(This is a 2D example, but I'm also interested in >2 dimensional cases)

EDIT: I know the p and q elements in the vectors must align with one another, but is that the only condition that’s required? (The loss of 2D structure is fine?)

• The 2d structure is just a matter of how you store the data. In the end, you need to loop through all the entries, do the element-wise calculations, and sum them. It doesn't matter if you store the values in a vector, matrix, or multi-dimensional array. In R in fact it would be the same calculation in each case sum(p * log(p/q)), no matter of their shape.
– Tim
May 17, 2021 at 19:05

$$D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$$
so to calculate it, you need to evaluate two probability distributions $$P$$ and $$Q$$ over the same values $$x$$. If you have the probabilities already calculated in a vector or a matrix, then yes, they need to be aligned.