1
$\begingroup$

I’m using a pretrained CNN to extract features from a set of medical images. These feature vectors are then used to train/test a SVM. The goal is to determine if disease is present in the images.

During my test phase, I’m achieving 100% mean per class accuracy. My ROC and precision/recall curves are perfect. My data size is n=193 by p=1536.

To determine if I’m doing something wrong, here is what I’ve done:

  1. 5 and 10-fold cross validation using pythons sklearn built in functions and examples for SVM. It was a grid search method that found the best hyperparamters for a linear and RBF kernel. After CV, the best kernel was a linear SVM.

  2. Leave one out CV with linear kernel. Achieved 100% accuracy.

  3. I checked my training and test accuracy vs. the training set size. What I find is that with a small training set, my test accuracy is low. As training set size increases, my test accuracy eventually achieves 100%.

  4. I double checked that I’m not accidentally mixing training/test data points.

  5. I also trained/tested a Random Forrest classifier and even with that method my results during testing were nearly 100%.

Perhaps the problem is too simple and obvious, but I guess what I’m trying to find out is what else can I do to make sure my results are real and convince others that it’s in fact real?

I have multiple datasets that I tried this on. I’ve got two normal datasets. One normal set come from a different sensor and the other one from the same sensor as my abnormal images. Could the abnormals be so obviously abnormal that it makes it simple to separate the two classes?

I’m not doing any kind of normalization before the SVM, subtraction of the mean set of feature vectors or anything like that. I figured if my results are so good as is, then there isn’t a need to do anything more. Thanks for any suggestions.

$\endgroup$
  • $\begingroup$ A few thoughts. 1) Is it possible that the images were used in the training of the CNN? 2) Is it possible that there are multiple images of the same culture (or whatever the images are of), and some made it into the training set, while others into the testing set? 3) Did you look at the feature importances reported by the RF? Are there any dominant features that could point to leakage? 4) Does it look like an easy problem if you just look at the images? Like, are all the diseased ones red, and all the healthy ones green, or something like that? $\endgroup$ – Matthew Drury Jun 10 '18 at 21:24
  • $\begingroup$ Thanks for the reply. 1) I used a pretrain network to extract features and didn’t fine tune the network. 2) I checked this several times over and no, this didn’t occur. 3) I did look at the importance features. Some features stuck out. I’m not sure what you mean by leakage? To what extent does a feature need to stick out to be considered leakage? 4) no, it’s not that easy. The images differ only in that one has an abnormal cysts near the liver and the other doesn’t. The cysts are of different sizes. Some are simple to see while others are not. Thanks for the help! $\endgroup$ – user9110057 Jun 11 '18 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.