I'm training a simple logistic regression classifier on top of a rich feature set of 512 features for a binary classification problem. The training set is 200 observations and the validation set is 50 observations. These sizes cannot be changed. Early stopping is used based on the validation set to prevent overfitting (which is of course highly likely on this small training dataset). There's a separate test set for evaluation afterwards. The datasets are fairly unbalanced with ~15% of observations being one class.
I'm experiencing a strange phenomena, where the validation loss will continuously go down during the optimization process, but the validation accuracy will worsen at the same time (i.e. also go down). Any suggestions for why this might happen?
The loss function is crossentropy.