I have a convolution neural network with random weights initialized and Trained to perform binary classification. I have 2000 images as training data and 2000 validation data. The problem I am trying to solve is if the image is healthy or not.The loss function used is categorical cross entropy.
I have implemented the model in tensorflow.(tf.nn.sparse_softmax_cross_entropy_with_logits) The validation loss for that model with random weights is 0.71 and an accuracy was 58%. However, the same architecture as above initialized with a pre-trained weights gives a loss of 0.79 and accuracy 65% (Both the models converged without overfitting)
Does any one have any idea on why the loss in the second case is high yet has a better accuracy compared to the first one?
To put it another way, what is the contributing factor to the discrepancy between classification accuracy and categorical cross entropy, where a higher cross entropy corresponds to a better classification accuracy?