Thanks for taking the time to read this I'm new to Machine Learning and so am going through a Kaggle competition to help me improve but I have a question. How can I compare different classifiers?? My Python isn't as neat as I would like it to be but i think it's correct. Please let me know if I'm doing something strange
train = pca(train) cfr = KNeighborsClassifier(n_neighbors=neighbours, algorithm="kd_tree") cv = cross_validation.KFold(len(train), n_folds=10, indices=True) results =  i = 0 count =0 for traincv, testcv in cv: ClassPred = cfr.fit(train[traincv], target[traincv]).predict(train[testcv]) for j in range(0,(len(train)/10)): labelindex = testcv[j] if (ClassPred[j] == target[labelindex]): i = i+1 accuracy = (i/2000.0)*100 i=0 results.append(accuracy) count = count + 1 print "accuracy for fold", count, " : ", accuracy,"%" print "time after fold", count, " : ", elspasedfold,"%" elapsed = (time.clock() - start) #print out the mean of the cross-validated results print "Results for RBF, c = 10.0, gamma=0.1 \n" + str(np.array(results).mean()) print "Time taken is %ds" % elapsed`.
As you can see I'm doing 10 fold cross validation on the training data and hopefully producing an 'accuracy' value out of the other end. My question here is how can I compare different classifiers?? If I choose cfr as a different classifier (e.g. Random Forest) and get a value for that how do i statistically compare the two?
My initial thoughts are to use a two-tailed t-test on the values for each fold (so 10 values per classifier) instead of just taking using the one value it currently outputs, which I think is an average of all the folds combined. I would need to then get a p-value to see if the differences are significant. I am not sure how I would go about implementing this however or if this is the correct thing to do. My stats is a little patchy (Which I'm working on) but any help anyone can give me would be much appreciated.
So to clarify, using python and sci-kit what would be the best way to compare the performance of different classifiers on my data? Thanks!
EDIT: The tutorial assessment criteria is that 10 fold cross validation must be used.