# Applying k-fold Cross validation over Training set and Test set with together (KNN Classification)

I am trying to find confusion matrix of Training set and Test set with together. But I can not understand that How I will fit this line clf.fit(X,y). I am using this clf.fit(X,y) to fit . Now I have a question : Is this method clf.fit(X,y) right ?

And How can I apply k-fold Cross validation over Training set and Test set with together ?

Here is my code:

# Import dataset
import pandas as pd
names = ['id', 'clump_thickness', 'uniformity_of_cell_size', 'uniformity_of_cell_shape', 'marginal_adhesion', 'single_epithial_cell_size', 'bare_nuclei', 'bland_chromatin', 'normal_nucleoli', 'mitoses', 'benign_malignant', '', '', '']
# names =  ['id', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', '', '', '']

# Split test and train data
import numpy as np
from sklearn.model_selection import train_test_split
X = np.array(dataset.ix[:, 1:10])
y = np.array(dataset['benign_malignant'])

# Applying k-fold Method
from sklearn.cross_validation import StratifiedKFold
kfold = 10 # no. of folds (better to have this at the start of the code)

skf = StratifiedKFold(y, kfold, random_state = 0)

# Stratified KFold: This first divides the data into k folds. Then it also makes sure that the distribution of the data in each fold follows the original input distribution
# Note: in future versions of scikit.learn, this module will be fused with kfold

skfind = [None]*len(skf) # indices
cnt=0
for train_index in skf:
skfind[cnt] = train_index
cnt = cnt + 1

# skfind[i][0] -> train indices, skfind[i][1] -> test indices
# Supervised Classification with k-fold Cross Validation

from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
import time

conf_mat = np.zeros((2,2)) # Initializing the Confusion Matrix

n_neighbors = 1; # better to have this at the start of the code

# 10-fold Cross Validation

for i in range(kfold):
train_indices = skfind[i][0]
test_indices = skfind[i][1]

clf = []
clf = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)
X_train = X[train_indices]
y_train = y[train_indices]
X_test = X[test_indices]
y_test = y[test_indices]

# Training -------------------------
# tic = time.time()
clf.fit(X,y) # here , I am confused. Is it correct ?
# toc = time.time()
# print ("training time= ", toc-tic) # roughly 2.5 secs

# Testing --------------------------
y_predict_train = []
y_predcit_test = []
# tic = time.time()
y_predict_train = clf.predict(X_train) # output is labels and not indices
y_predict_test = clf.predict(X_test)
# toc = time.time()
# print ("testing time = ", toc-tic) # roughly 0.3 secs

# Compute confusion matrix
cm1 = []
cm2 = []
cm1 = confusion_matrix(y_train,y_predict_train)
cm2 = confusion_matrix(y_test,y_predict_test)
print(cm1)
print(cm2)
# conf_mat = conf_mat + cm

• It is unclear to me what you mean by "And How can I apply k-fold Cross validation over Training set and Test set with together ?".
– Gijs
Commented Aug 23, 2017 at 8:27
• Is there any way to find confusion_matrix of training set and test set together ? @Gijs Commented Aug 23, 2017 at 8:31

You are fitting with the whole set, after constructing a train and test set. Instead, use train_test_split, as indicated in http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation.

The usual approach is to fit on the training set, and then compare the predictions on the test set ('X_test') with the true values on the test set (y_test).

clf.fit(X_train, y_train)
predictions = clf.predict(X_test)

from sklearn.metrics import confusion_matrix

confusion_matrix(predictions, y_test)


To directly answer your first question, if, instead, you use clf.fit(X, y), that should work and trains a classifier on the whole dataset.

EDIT

To get a full confusion matrix, apply the classifier to the whole dataset like this

clf.fit(X, y)
predictions = clf.predict(X)
confusion_matrix(y, predictions)

• This confusion_matrix is which for ? Training or Test set ?? Commented Aug 23, 2017 at 8:33
• I want to find confusion_matrix both Training set and Test set together.@Gijs Commented Aug 23, 2017 at 8:39