Validation set early stopping on custom metric I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain.
As a concrete example, I am expecting to evaluate my model on a test set using the AUROC (Area-under-the-ROC-curve). So it seems reasonable to stop training a model if the improvement under this metric stops on the validation set. 
However, empirically I observe that using this strategy leads to oscillating training/validation curves, which can mis-lead an early stopping algorithm which stops training when there is no improvement for, say 10, epochs. 
On the other hand, using a mathematically simpler metric like the logistic loss shows a smooth decrease of the loss over >100 epochs. 
I am wondering whether the model could overfit under one metric, but not on the other, which seems counterintuitive. Maybe the changes under the model are just not relevant anymore under the real (domain-relevant) metric?
 A: For your first question:

I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain.

Early stopping should be implemented on a validation metric. If you want to evaluate your model on the AUC, it makes sense monitoring the same metric on the validation set.
However, you should be careful on one thing. It is possible for your model to overfit on the validation set! This is most relevant if you checkpoint your model at the epoch that achieved the best performance on a validation metric. Especially in models whose performance fluctuates heavily from epoch to epoch, this performance's could have been the result of chance (the weights updated in such a way that they got right the most examples in the validation set). If this is the case, this isn't indicative of the generalization capability of the model. For this reason you should have both a validation and a test set.

I am wondering whether the model could overfit under one metric, but not on the other, which seems counterintuitive. Maybe the changes under the model are just not relevant anymore under the real (domain-relevant) metric?

Generally, no. But some metrics could better represent the overfitting than others. Stick with  the metric that you feel makes more sense in your domain. A rule of the thumb is that you can always print a confusion matrix and see what classes your model predicts and what it gets wrong.
