Why ensemble of many deep-learning models did not work?

Question

I am trying to solve an image classification problem using DL, Keras and tensorflow. I added several layers of conv2D followed by batchnorm, pooling and dropout. I get a good accuracy ~95% with this. Now to improve it further, I thought that why not create many models (say 100 models based on same training and validation data. The only thing changed is the training data where the different kind of random transformations are applied ).

Once i have all the model, then get the majority voting on the predictions from all the models.

I see that the ensemble model performs worse on holdout data (opposite of my gut feeling) in most cases. I also tried bring diversity into the models by changing the network architecture and tuned parameters but none helped.

What may be the reason of the ensemble not working better that an individual model.

Answer 1

This heavily depends on the new 100 models that you have created. In general it is better to look at the correlation of the results inferred by your model. As similar results given by the models will bias the prediction coming from the voting.