I've trained a network using a genetic algorithm and I have two possible fitness functions for my GA: MSE and CErr.
If I use MSE as my fitness function, over time MSE decreases and classification error decreases. Looking at my plots, it looks like MSE and CErr are directly related. However, if I use CErr as my fitness function, over time my CErr decreases but my MSE doesn't always decrease.
For example, consider these two training attempts, the first with MSE as fitness function and second with CErr as fitness function. The data, train/test split, activation function, etc everything is equal except fitness function. Accuracy is just a function of classification error (
1 - CErr = Acc).
From this I can deduce a lower MSE means a lower CErr but a lower CErr doesn't always mean a lower MSE. For a genetic algorithm, I should use MSE as my fitness function because if MSE were to rise to reach a better CErr, it might get trapped in local minima as better solutions are guaranteed when the MSE is lower (and the algorithm, through crossover and mutation, will focus on areas in the search volume which are suboptimal).
But why do I get results like this in the first place? I thought MSE was directly proportional to CErr and vice-versa but I do not always observe this.