I’m using a pretrained CNN to extract features from a set of medical images. These feature vectors are then used to train/test a SVM. The goal is to determine if disease is present in the images.
During my test phase, I’m achieving 100% mean per class accuracy. My ROC and precision/recall curves are perfect. My data size is n=193 by p=1536.
To determine if I’m doing something wrong, here is what I’ve done:
5 and 10-fold cross validation using pythons sklearn built in functions and examples for SVM. It was a grid search method that found the best hyperparamters for a linear and RBF kernel. After CV, the best kernel was a linear SVM.
Leave one out CV with linear kernel. Achieved 100% accuracy.
I checked my training and test accuracy vs. the training set size. What I find is that with a small training set, my test accuracy is low. As training set size increases, my test accuracy eventually achieves 100%.
I double checked that I’m not accidentally mixing training/test data points.
I also trained/tested a Random Forrest classifier and even with that method my results during testing were nearly 100%.
Perhaps the problem is too simple and obvious, but I guess what I’m trying to find out is what else can I do to make sure my results are real and convince others that it’s in fact real?
I have multiple datasets that I tried this on. I’ve got two normal datasets. One normal set come from a different sensor and the other one from the same sensor as my abnormal images. Could the abnormals be so obviously abnormal that it makes it simple to separate the two classes?
I’m not doing any kind of normalization before the SVM, subtraction of the mean set of feature vectors or anything like that. I figured if my results are so good as is, then there isn’t a need to do anything more. Thanks for any suggestions.