I'm quite new to machine learning so sorry if this is a simple question. When constructing a regression model (in the form of a neural network), we might use MSE as the main metric of comparison between models. However, when constructing classification models, where the outputs are encoded as one-hot target vectors, cross entropy is a better loss function to use for training, and we would generally assess models by accuracy/precision/recall metrics. Surely though the final crossentropy loss for a trained model would still tell you how well different models are capable of fitting the data, to point one's hyperparameter search in the right direction? Are there any caveats to be aware of in interpreting crossentropy loss for a trained model and using it to compare between different neural networks?
We use cross-entropy for training a model but not for comparing different models. Cross-entropy is a loss function, which is a function for model parameters.
On the other hand, accuracy/precision/recall etc are model independent.