I'm using the VGG19 convolutional network for image recognition (to be specific, the keras implementation). I've downloaded the ILSVRC2012 validation dataset and the ground truth from here, to check if the accuracy of VGG19 matches my expectation. Note that the labeling of the linked ground truth is different from the default ground truth labeling provided by ImageNet.

Since VGG19 expects input of the shape (224, 224, 3), one has to choose a way of dealing with input which is not of square shape. I avoided naive resizing which violates the aspect ratio (because it might change the ground truth). What I did is the following:

  1. Filter the validation dataset such that only images remain which satisfy 0.95 <= height / width <= 1.05 (i.e. which are nearly of square shape). Almost 4000 images fulfill this property.

  2. resize those images to 224 × 224 and call this smaller dataset D.

  3. a) Evaluate (Top 1 classification) VGG19 on D, which results in an accuracy of ~ 59.48%
    b) Preprocess every image in D using keras.applications.vgg19.preprocess_input, call that new dataset P and evaluate (Top 1 classification) VGG19 on P, which results in an accuracy of ~ 52.46%

So now my two questions are:

  1. Why is the performance of VGG19 significantly lower than ~ 75%, which is the Top 1 accuracy on the validation dataset the authors of VGG claim to achieve?
  2. Why does preprocessing lower the accuracy even further?

Since D only contains the 4000 nearly square shaped images, a bias might explain the overall bad performance. To check this, I center-cropped every image from the validation dataset, resized it to 224 × 224 and repeated the evaluation on those – with similar results (~ 60% vs ~ 54%).

My setup: Keras 2.2.2, Tensorflow-GPU 1.10, Ubuntu 16.04

  • $\begingroup$ @AlexR. As you mentioned, I took the validation dataset, rescaled every image's smaller side to 256, took a central 224 × 224 crop and then evaluated VGG19 on those images. The result is 64.37%, which is still far from 75%. $\endgroup$ – user3389669 Aug 27 '18 at 14:41

I guess I found one error: I loaded the validation images via OpenCV (cv2 module) by executing cv2.imread, which returns a Numpy array, where the color channel ordering is BGR. Using keras.applications.vgg19.preprocess_input on that Numpy array accidentally reverts the channel ordering back to RGB (although the target ordering is BGR) and then subtracts the mean BGR pixel [103.939, 116.779, 123.68].

After realizing that, I performed the following evaluation:

  1. Center-crop each image of the validation dataset using a box of maximal size
  2. Scale the cropped images to 224 × 224 using cv2.resize(..., interpolation = cv2.INTER_LANCZOS4)
  3. a) Evaluate VGG19 on the non-centered batch -> accuracy of ~ 63.25%

    b) Center the pixel color values of the batch by image_batch = np.subtract(image_batch, [103.939, 116.779, 123.68], dtype = np.float32) and evaluate VGG19 on that -> accuracy of ~ 70.21%

This is now way closer to the expected 75% accuracy. The gap might be credited to the way I evaluate VGG (single center crop vs multi-crop/dense evaluation an the paper) but I'd like to ask you guys:

Did I miss an additional preprocessing step?

A side note: It's worth writing

image_batch = np.subtract(image_batch, [103.939, 116.779, 123.68], dtype = np.float32)

instead of just

image_batch -= [103.939, 116.779, 123.68]

because the latter one results on my machine in a float64 array, which is some kind of overkill (in terms of size and GPU performance).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.