I am using efficient-net to classify images. I have trained model successfully and wanted to quantize it using tf-lite. I tried all the methods available in tf-lite quantization to check accuracy, size and latency. Size is decreased as per docs x4 times and little change in accuracy but my problem is the latency is increased drastically but as per documentation latency is also supposed to decrease.

I am using google colab, tried in both CPU and gpu mode with tf version 2.3.0. Running inference in original model for test images is about 10sec but for same test images(around 193 images on test set) for quantized model takes 300 sec to run. Tried different batch of test data, the time to load the data is approx. 5sec.

Here is my code:

def repr_gen_data():
  a = []
  for image_path in test_image_list[:100]:
    image = cv2.imread(image_path,cv2.IMREAD_COLOR)
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    image = cv2.resize(image, (IMG_SIZE,IMG_SIZE))
    image_ = image.astype(np.float32)
    # image_ = tf.convert_to_tensor(image, dtype=tf.uint8)

  a = np.array(a) 
  for i in tf.data.Dataset.from_tensor_slices(a).batch(1).take(100):
    yield [i] 

converter = tf.lite.TFLiteConverter.from_saved_model(model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = tf.lite.RepresentativeDataset(rep_data_gen)

converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]

tflite_model = converter.convert()

import pathlib
tflite_models_dir = pathlib.Path(quantized_dir)
tflite_models_dir.mkdir(exist_ok=True, parents=True)
tflite_model_file = tflite_models_dir/"model_fullInt_cpu_2_v03.tflite"

Inference: test_data is obtained from generator

test_batch = test_data.cache().batch(1).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)

interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

prediction = []
test_labels = []

start_time = time.time()
for img, label in test_batch.take(193):
  interpreter.set_tensor(input_details[0]['index'], img)

  test_pred = tf.argmax(interpreter.get_tensor(output_details[0]['index']), axis=1)
  # output = interpreter.tensor(output_details)
  # print(test_pred)
  # break

I could not trace what might be the problem, I went through the issue of github in tensorflow's repo, there they have mentioned that this is optimized only for mobile cpu, is this the case ? But the docs doesn't mention anything like that. I had feeling that my data loader is increasing the latency, but checked that to load the test data with different batches takes approx. 5sec.


-post training quantization tensorflow docs


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.