So, I am trying to understand if I have fair split of my train and val sets using train_test_split of sklearn, so I decided to run the KL divergence and JS div tests and I get the following results. How can I fix this or how else can I do train_test_split if this is not correct for my continuous values target y?

x_train, x_val, y_train, y_val=train_test_split(x,y,test_size=0.3, random_state=42)

p = np.array([6.33527306, 0.17195741, 0.01810078, 0.01810078])
q = np.array([7.36958404, 0.09665028, 0.02416257, 0.02416257])

probs_train = plt.hist(y_train, density=True, bins=4)
probs_val = plt.hist(y_val, density=True, bins=4)

and I have the p and q from the first arrays provided by probs_train and probs_val:

probs train: (array([6.33527306, 0.17195741, 0.01810078, 0.01810078]), array([0.217 , 0.369825, 0.52265 , 0.675475, 0.8283 ]), <BarContainer object of 4 artists>)


probs val: (array([7.36958404, 0.09665028, 0.02416257, 0.02416257]), array([0.239 , 0.372075, 0.50515 , 0.638225, 0.7713 ]), <BarContainer object of 4 artists>)

def KL_div(p, q):
        return sum(p[i] * log2(p[i]/q[i]) for i in range(len(p)))

print("KL(p, q): ", KL_div(p, q))
print("KL(q, p): ", KL_div(q, p))
print("p: ", p)
print("q: ", q)

js_pq = jensenshannon(p, q, base=2)
print('JS(P || Q) Distance: %.3f' % js_pq)
js_qp = jensenshannon(q, p, base=2)
print('JS(Q || P) Distance: %.3f' % js_qp)

Results are:

p: [6.33527306 0.17195741 0.01810078 0.01810078]
q: [7.36958404 0.09665028 0.02416257 0.02416257]

KL(p, q): -1.2543610466991473
KL(q, p): 1.547671142537377

JS(P || Q) Distance: 0.042
JS(Q || P) Distance: 0.042

My y_train and y_val histogram looks like this: enter image description here

and the normalized histogram looks like this: enter image description here

Also, please note that 4 was the largest number I could pick for bin number so that KL divergence won't throw NAN due to some bins being zero.

Here's the histogram of y itself:

enter image description here

  • 1
    $\begingroup$ Estimating KL divergence between two continuous-valued distributions from samples can be negative. That’s normal. What do you want to “fix”? $\endgroup$ Nov 25 '21 at 4:01
  • $\begingroup$ @AryaMcCarthy thanks for looking at the question. So, how do you interpret the result of this KL(p,q) and KL(q,p)? is it in your opinion a good split for train and val using train_test_split? $\endgroup$
    – Mona Jalal
    Nov 25 '21 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.