0
$\begingroup$

I am trying to find confusion matrix of Training set and Test set with together. But I can not understand that How I will fit this line clf.fit(X,y). I am using this clf.fit(X,y) to fit . Now I have a question : Is this method clf.fit(X,y) right ?

And How can I apply k-fold Cross validation over Training set and Test set with together ?

Here is my code:

# Import dataset
import pandas as pd
names = ['id', 'clump_thickness', 'uniformity_of_cell_size', 'uniformity_of_cell_shape', 'marginal_adhesion', 'single_epithial_cell_size', 'bare_nuclei', 'bland_chromatin', 'normal_nucleoli', 'mitoses', 'benign_malignant', '', '', '']
# names =  ['id', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', '', '', '']
dataset = pd.read_csv("Breast-Cancer-Preprocessed.csv", header = None, names = names)

# Split test and train data
import numpy as np
from sklearn.model_selection import train_test_split
X = np.array(dataset.ix[:, 1:10])
y = np.array(dataset['benign_malignant'])

# Applying k-fold Method
from sklearn.cross_validation import StratifiedKFold
kfold = 10 # no. of folds (better to have this at the start of the code)

skf = StratifiedKFold(y, kfold, random_state = 0)

# Stratified KFold: This first divides the data into k folds. Then it also makes sure that the distribution of the data in each fold follows the original input distribution 
# Note: in future versions of scikit.learn, this module will be fused with kfold

skfind = [None]*len(skf) # indices
cnt=0
for train_index in skf:
    skfind[cnt] = train_index
    cnt = cnt + 1

# skfind[i][0] -> train indices, skfind[i][1] -> test indices
# Supervised Classification with k-fold Cross Validation

from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
import time

conf_mat = np.zeros((2,2)) # Initializing the Confusion Matrix

n_neighbors = 1; # better to have this at the start of the code

# 10-fold Cross Validation


for i in range(kfold):
    train_indices = skfind[i][0]
    test_indices = skfind[i][1]

    clf = []
    clf = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)
    X_train = X[train_indices]
    y_train = y[train_indices]
    X_test = X[test_indices]
    y_test = y[test_indices]

    # Training -------------------------
    # tic = time.time()
    clf.fit(X,y) # here , I am confused. Is it correct ? 
    # toc = time.time()
    # print ("training time= ", toc-tic) # roughly 2.5 secs

    # Testing --------------------------
    y_predict_train = []
    y_predcit_test = []
    # tic = time.time()
    y_predict_train = clf.predict(X_train) # output is labels and not indices
    y_predict_test = clf.predict(X_test)
    # toc = time.time()
    # print ("testing time = ", toc-tic) # roughly 0.3 secs

    # Compute confusion matrix
    cm1 = []
    cm2 = []
    cm1 = confusion_matrix(y_train,y_predict_train)
    cm2 = confusion_matrix(y_test,y_predict_test)
    print(cm1)
    print(cm2)
    # conf_mat = conf_mat + cm 
$\endgroup$
2
  • $\begingroup$ It is unclear to me what you mean by "And How can I apply k-fold Cross validation over Training set and Test set with together ?". $\endgroup$
    – Gijs
    Commented Aug 23, 2017 at 8:27
  • $\begingroup$ Is there any way to find confusion_matrix of training set and test set together ? @Gijs $\endgroup$ Commented Aug 23, 2017 at 8:31

1 Answer 1

1
$\begingroup$

You are fitting with the whole set, after constructing a train and test set. Instead, use train_test_split, as indicated in http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation.

The usual approach is to fit on the training set, and then compare the predictions on the test set ('X_test') with the true values on the test set (y_test).

clf.fit(X_train, y_train)
predictions = clf.predict(X_test)

from sklearn.metrics import confusion_matrix

confusion_matrix(predictions, y_test)

To directly answer your first question, if, instead, you use clf.fit(X, y), that should work and trains a classifier on the whole dataset.

EDIT

To get a full confusion matrix, apply the classifier to the whole dataset like this

clf.fit(X, y)
predictions = clf.predict(X)
confusion_matrix(y, predictions)
$\endgroup$
2
  • $\begingroup$ This confusion_matrix is which for ? Training or Test set ?? $\endgroup$ Commented Aug 23, 2017 at 8:33
  • $\begingroup$ I want to find confusion_matrix both Training set and Test set together.@Gijs $\endgroup$ Commented Aug 23, 2017 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.