So, I am trying to understand if I have fair split of my train and val sets using train_test_split of sklearn, so I decided to run the KL divergence and JS div tests and I get the following results. How can I fix this or how else can I do train_test_split if this is not correct for my continuous values target y?
x_train, x_val, y_train, y_val=train_test_split(x,y,test_size=0.3, random_state=42) p = np.array([6.33527306, 0.17195741, 0.01810078, 0.01810078]) q = np.array([7.36958404, 0.09665028, 0.02416257, 0.02416257]) probs_train = plt.hist(y_train, density=True, bins=4) probs_val = plt.hist(y_val, density=True, bins=4)
and I have the
q from the first arrays provided by
probs train: (array([6.33527306, 0.17195741, 0.01810078, 0.01810078]), array([0.217 , 0.369825, 0.52265 , 0.675475, 0.8283 ]), <BarContainer object of 4 artists>)
probs val: (array([7.36958404, 0.09665028, 0.02416257, 0.02416257]), array([0.239 , 0.372075, 0.50515 , 0.638225, 0.7713 ]), <BarContainer object of 4 artists>)
def KL_div(p, q): return sum(p[i] * log2(p[i]/q[i]) for i in range(len(p))) print("KL(p, q): ", KL_div(p, q)) print("KL(q, p): ", KL_div(q, p)) print("p: ", p) print("q: ", q) js_pq = jensenshannon(p, q, base=2) print('JS(P || Q) Distance: %.3f' % js_pq) js_qp = jensenshannon(q, p, base=2) print('JS(Q || P) Distance: %.3f' % js_qp)
p: [6.33527306 0.17195741 0.01810078 0.01810078] q: [7.36958404 0.09665028 0.02416257 0.02416257] KL(p, q): -1.2543610466991473 KL(q, p): 1.547671142537377 JS(P || Q) Distance: 0.042 JS(Q || P) Distance: 0.042
Also, please note that 4 was the largest number I could pick for bin number so that KL divergence won't throw NAN due to some bins being zero.
Here's the histogram of y itself: