Issues in pattern recognition and plots

Question

I am trying to compare the performance of a classification result with Bayes classifier, K-NN, principal component analysis (PCA).

I have doubts regarding the following (please excuse my lack of programming skills since I am a biologist and not a programmer thus finding hard to follow the matlab documentation)

1. In the Matlab code

   Class = knnclassify(Sample, Training, Group, k)
   Group =  [1;2;3]   //where 1,2,3 represents Class A,B,C respectively.
   

   What goes in the sample because my data is a 100 row 1 column for each of the classes? So Group 1 contains data like $[0.9;0.1;......n]$ where $n=100$. Would the sample be a vector containing random mixtures of the data points from the three classes? Same query/doubt for Training matrix.

2. I cannot follow crossval() & cvpartition() function given in MATLAB documentation crossval(). Would be obliged if a simpler version of it is provided here.

3. Is there a code for multi-class ROC since Internet resources provide information about binary class ROC. How to proceed for ROC plot when there are three or more classes like in my case?

4. Is it possible to obtain a surface plot from the confusion mmatrix from the command confusemat()?

Answer 1

I did a webinar titled An Introduction to Classification with MATLAB. You can download the code and the dataset from the MATLAB file exchange:

http://www.mathworks.com/matlabcentral/fileexchange/28770-introduction-to-classification

I'm attaching some code directly that might be helpful

%%  Use a Naive Bayes Classifier to develop a classification model

% Some of the features exhibit significant correlation, however, its
% unclear whether the correlated features will be selected for our model

% Start with a Naive Bayes Classifier

% Use cvpartition to separate the dataset into a test set and a training set
% cvpartition will automatically ensure that feature values are evenly
% divided across the test set and the training set

% Create a cvpartition object that defined the folds
c = cvpartition(Y,'holdout',.2);

% Create a training set

X_Train = X(training(c,1),:);
Y_Train = Y(training(c,1));

%%  Train a Classifier using the Training Set

Bayes_Model = NaiveBayes.fit(X_Train, Y_Train, 'Distribution','kernel');

%%  Evaluate Accuracy Using the Test Set

clc

% Generate a confusion matrix
[Bayes_Predicted] = Bayes_Model.predict(X(test(c,1),:));
[conf, classorder] = confusionmat(Y(test(c,1)),Bayes_Predicted);
conf

% Calculate what percentage of the Confusion Matrix is off the diagonal
Bayes_Error = 1 - trace(conf)/sum(conf(:))


%%  Naive Bayes Classification using Forward Feature Selection

% Create a cvpartition object that defined the folds
c2 = cvpartition(Y,'k',10);

% Set options
opts = statset('display','iter');

fun = @(Xtrain,Ytrain,Xtest,Ytest)...
      sum(Ytest~=predict(NaiveBayes.fit(Xtrain,Ytrain,'Distribution','kernel'),Xtest));

[fs,history] = sequentialfs(fun,X,Y,'cv',c2,'options',opts)
White_Wine.Properties.VarNames(fs)

Ad in an illustration of how to calculate an ROC curve. Please note: this example is using a bagged decision tree rather than a Naive Bayes classifier

%%  Run Treebagger Using Sequential Feature Selection
tic
f = @(X,Y)oobError(TreeBagger(50,X,Y,'method','classification','oobpred','on'),'mode','ensemble');
opt = statset('display','iter');
[fs,history] = sequentialfs(f,X,Y,'options',opt,'cv','none');
toc
%%  Evaluate the accuracy of the model using a performance curve

Test_Results = dataset(Y_Test, Predicted, Class_Score);
[xVal,yVal,~,auc] = perfcurve(Test_Results.Predicted, ...
    Test_Results.Class_Score(:,4),'6'); 

plot(xVal,yVal)
xlabel('False positive rate'); ylabel('True positive rate')

Answer 2

I would like to answer only first one. Going backward with parameters help.

k is number of points used in classification/regression. An example would be helpful here. Suppose you would like to sell your house, you nearest neighbor in street sold her house for 250K. Second nearest neighbor sold her house for 150K and third nearest neighbor for 200K. What value should you use?

if k = 1, you use 250. if k = 2 , you use ( 250 + 150 ) / 2 if k = 3 , you use (250 + 150 + 200) / 3.

In classification, you use majority vote for this purpose.

Group is number of classes , you would like to classify. It should be fixed number.

Training is your data, you collect. As you asked it should be mixture of groups.

Sample is the data which will be classified according to nearest neighbor method described above. It should not be mixture. You may only give data from group 1. But normally you do not know which groups, your data points belongs.