Applying k-fold Cross validation over Training set and Test set with together (KNN Classification)

Question

I am trying to find confusion matrix of Training set and Test set with together. But I can not understand that How I will fit this line clf.fit(X,y). I am using this clf.fit(X,y) to fit . Now I have a question : Is this method clf.fit(X,y) right ?

And How can I apply k-fold Cross validation over Training set and Test set with together ?

Here is my code:

# Import dataset
import pandas as pd
names = ['id', 'clump_thickness', 'uniformity_of_cell_size', 'uniformity_of_cell_shape', 'marginal_adhesion', 'single_epithial_cell_size', 'bare_nuclei', 'bland_chromatin', 'normal_nucleoli', 'mitoses', 'benign_malignant', '', '', '']
# names =  ['id', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', '', '', '']
dataset = pd.read_csv("Breast-Cancer-Preprocessed.csv", header = None, names = names)

# Split test and train data
import numpy as np
from sklearn.model_selection import train_test_split
X = np.array(dataset.ix[:, 1:10])
y = np.array(dataset['benign_malignant'])

# Applying k-fold Method
from sklearn.cross_validation import StratifiedKFold
kfold = 10 # no. of folds (better to have this at the start of the code)

skf = StratifiedKFold(y, kfold, random_state = 0)

# Stratified KFold: This first divides the data into k folds. Then it also makes sure that the distribution of the data in each fold follows the original input distribution 
# Note: in future versions of scikit.learn, this module will be fused with kfold

skfind = [None]*len(skf) # indices
cnt=0
for train_index in skf:
    skfind[cnt] = train_index
    cnt = cnt + 1

# skfind[i][0] -> train indices, skfind[i][1] -> test indices
# Supervised Classification with k-fold Cross Validation

from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
import time

conf_mat = np.zeros((2,2)) # Initializing the Confusion Matrix

n_neighbors = 1; # better to have this at the start of the code

# 10-fold Cross Validation


for i in range(kfold):
    train_indices = skfind[i][0]
    test_indices = skfind[i][1]

    clf = []
    clf = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)
    X_train = X[train_indices]
    y_train = y[train_indices]
    X_test = X[test_indices]
    y_test = y[test_indices]

    # Training -------------------------
    # tic = time.time()
    clf.fit(X,y) # here , I am confused. Is it correct ? 
    # toc = time.time()
    # print ("training time= ", toc-tic) # roughly 2.5 secs

    # Testing --------------------------
    y_predict_train = []
    y_predcit_test = []
    # tic = time.time()
    y_predict_train = clf.predict(X_train) # output is labels and not indices
    y_predict_test = clf.predict(X_test)
    # toc = time.time()
    # print ("testing time = ", toc-tic) # roughly 0.3 secs

    # Compute confusion matrix
    cm1 = []
    cm2 = []
    cm1 = confusion_matrix(y_train,y_predict_train)
    cm2 = confusion_matrix(y_test,y_predict_test)
    print(cm1)
    print(cm2)
    # conf_mat = conf_mat + cm 

Answer 1

You are fitting with the whole set, after constructing a train and test set. Instead, use train_test_split, as indicated in http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation.

The usual approach is to fit on the training set, and then compare the predictions on the test set ('X_test') with the true values on the test set (y_test).

clf.fit(X_train, y_train)
predictions = clf.predict(X_test)

from sklearn.metrics import confusion_matrix

confusion_matrix(predictions, y_test)

To directly answer your first question, if, instead, you use clf.fit(X, y), that should work and trains a classifier on the whole dataset.

EDIT

To get a full confusion matrix, apply the classifier to the whole dataset like this

clf.fit(X, y)
predictions = clf.predict(X)
confusion_matrix(y, predictions)