This question already has an answer here:
- Analysis of Kullback-Leibler divergence 3 answers
In trying to understand the Kullback-Leibler Divergence I conceive it as a metric that if minimized would make the Approximation PDF Q as close as possible to the True PDF P either in the neighborhoods of X where P places the highest probability mass (K(P||Q)) or in the neighborhoods of X where Q places the highest probability mass (K(Q||P)). So it is like saying to Q "try your best to approximate P where P thinks are the most probable neighborhoods" or inversely "try to approximate P the best in the neighborhoods you believe are the most probable."
On the other hand, trying to arrive to an intuitive understanding of the Kolmogorov-Smirnov statistic, I view it as a decision to underplay local divergences between two probability distributions functions in favor of the 'larger picture' which is reflected in the divergence of their respective Cumulative Distributions.
I wonder if these thoughts make sense, and how you would like to add to them with your own intuitive interpretations of the two Divergence 'metrics'.
In a more practical level when and how do you believe Kullback-Leibler Divergence and Kolmogorov-Smirnov statistic should be used?
A case in hand might be to compare the predicted probabilities by a classifier --say Logistic Regression-- to the empirical ones using the Validation set. Consequently compute the divergence between the two distributions --predicted and empirical-- using an appropriate divergence 'metric'. Finally use this information to select a model that best approximates the conditional probability P(Y|X). Which metric should we use and why?