3
$\begingroup$

I've trained a network using a genetic algorithm and I have two possible fitness functions for my GA: MSE and CErr.

If I use MSE as my fitness function, over time MSE decreases and classification error decreases. Looking at my plots, it looks like MSE and CErr are directly related. However, if I use CErr as my fitness function, over time my CErr decreases but my MSE doesn't always decrease.

For example, consider these two training attempts, the first with MSE as fitness function and second with CErr as fitness function. The data, train/test split, activation function, etc everything is equal except fitness function. Accuracy is just a function of classification error (1 - CErr = Acc).

enter image description here

enter image description here

From this I can deduce a lower MSE means a lower CErr but a lower CErr doesn't always mean a lower MSE. For a genetic algorithm, I should use MSE as my fitness function because if MSE were to rise to reach a better CErr, it might get trapped in local minima as better solutions are guaranteed when the MSE is lower (and the algorithm, through crossover and mutation, will focus on areas in the search volume which are suboptimal).

But why do I get results like this in the first place? I thought MSE was directly proportional to CErr and vice-versa but I do not always observe this.

$\endgroup$

1 Answer 1

3
+50
$\begingroup$

If your algorithm is estimating probabilities of class membership, then mean-square error (MSE) is effectively the Brier score, a strictly proper scoring rule that is optimized by a model that provides the true probabilities.

Classification error (CErr) involves an often hidden assumption about assignment of class membership based on class probabilities. For example, in binary classification the model's class assignment would usually by default go to the class that has highest probability, a cutoff probability of 0.5. But if you choose such a cutoff in practice then you are effectively assuming that all mis-classifications have the same cost. If false negatives are more costly than false positives, however, then a different cutoff that lowers the false-negative rate might be more appropriate to minimize overall cost.

So there is no reason to expect that MSE and CErr would track together. Your criterion should be MSE or some other proper scoring rule to model class-membership probabilities first. Then use your relative costs of mis-classification to determine class assignments based on those probabilities.

$\endgroup$
3
  • $\begingroup$ Interesting. My network does consider, in the case of a binary classification problem, a output of 0.51 for one class as belonging to that class. I guess even if CErr is optimized, CErr is not a suitable measure because it doesn't show confidence in classification. If an CErr optimized network only cared about cutoffs of probabilities, an MSE optimized network might not and my neural outputs might be closer to 1.00 than the CErr optimized network, thus being more confident in classification. $\endgroup$
    – gator
    Apr 18, 2020 at 0:34
  • $\begingroup$ To make what I said short, my neural outputs in the CErr optimized network might be within the range [0.51, 1.00] whereas the MSE optimized network might be [0.90, 1.00] to give an example. $\endgroup$
    – gator
    Apr 18, 2020 at 0:35
  • 1
    $\begingroup$ @gator there are many ways that MSE and CErr as criteria can lead to different models. If you have pre-specified the cost tradeoff between false-negative and false-positive classifications you can consider a wide range of proper scoring rules to minimize expected cost in your application. This thread and this answer provide more detail and links to further reading. $\endgroup$
    – EdM
    Apr 18, 2020 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.