I have two questions (they can be solved by one answer). They are related to comparing the training metrics vs the validation metrics for each iteration / levels of regularization parameters. I am sorry for posting a long message but I am trying to explain my thoughts as clear as I can.

Generally from what I know we want to stop training when the validation metrics reach their max values (thus the so-called callback functions that are built-in in so many algorithms).

Question number 1: What happens if the validation metrics plateau? Should I relax my regularization parameters a bit because I am underfitting? (as to allow for a U shape in the validation metrics graph? U for logloss inverse U for AUC etc.) What if it still is stuck in the same value?. In that case what happens if I keep training without reaching an outrageous gap? This is closely related to my second question:

Question number 2: We generally want our training and validation metrics to hold relatively the same values, is that correct? However, what happens if for some reason the validation set and the training set are not representive of each other 100%? More specifically can there be some cases where the model is trained correctly (thus higher training metrics) and not overfit (i am not talking about huge gaps), while these cases are missing in the validation set (thus the validation metrics plateau)?I am asking because In real world, sometimes it is hard to gather huge amounts of data, thus some cases might not be well-represented. So even by performing k-fold validation n times, I may still be depending on the luck of the split; enough cases in the training set and enough of them in the test set. By keeping a very close gap in this situation I am running the risk of underfitting my model, is that correct?

If my way of thinking is not flawed, would a good solution be to allow for slightly higher metrics in the training set?

PS. I do not want to touch the test-set yet.

I have two questions (they can be solved by one answer). They are related to comparing the training metrics vs the validation metrics for each iteration / levels of regularization parameters. I am sorry for posting a long message but I am trying to explain my thoughts as clear as I can.

Generally from what I know we want to stop training when the validation metrics reach their min/max values (thus the so-called callback functions that are built-in in so many algorithms).

Question number 1: What happens if the validation metrics plateau? Should I relax my regularization parameters a bit because I am underfitting? (as to allow for a U shape in the validation metrics graph? U for logloss inverse U for AUC etc.) What if it still is stuck in the same value?. In that case what happens if I keep training without reaching an outrageous gap? This is closely related to my second question:

Question number 2: We generally want our training and validation metrics to hold relatively the same values, is that correct? However, what happens if for some reason the validation set and the training set are not representive of each other 100%? More specifically can there be some cases where the model is trained correctly (thus higher training metrics) and not overfit (i am not talking about huge gaps), while these cases are missing in the validation set (thus the validation metrics plateau)?I am asking because In real world, sometimes it is hard to gather huge amounts of data, thus some cases might not be well-represented. So even by performing k-fold validation n times, I may still be depending on the luck of the split; enough cases in the training set and enough of them in the test set. By keeping a very close gap in this situation I am running the risk of underfitting my model, is that correct?

If my way of thinking is not flawed, would a good solution be to allow for slightly higher metrics in the training set?

PS. I do not want to touch the test-set yet.

I have two questions (they can be solved by one answer). They are related to comparing the training metrics vs the validation metrics for each iteration / levels of regularization parameters. I am sorry for posting a long message but I am trying to explain my thoughts as clear as I can.

Generally from what I know we want to stop training when the validation metrics reach their max values (thus the so-called callback functions that are built-in in so many algorithms).

Question number 1: What happens if the validation metrics plateau? Should I relax my regularization parameters a bit because I am underfitting? (as to allow for a U shape in the validation metrics graph? U for logloss inverse U for AUC etc.). If not, what happens if I keep training without reaching an outrageous gap? This is closely related to my second question:

Question number 2: We generally want our training and validation metrics to hold relatively the same values, is that correct? However, what happens if for some reason the validation set and the training set are not representive of each other 100%? More specifically can there be some cases where the model is trained correctly (thus higher training metrics) and not overfit (i am not talking about huge gaps), while these cases are missing in the validation set (thus the validation metrics plateau)?I am asking because In real world, sometimes it is hard to gather huge amounts of data, thus some cases might not be well-represented. So even by performing k-fold validation n times, I may still be depending on the luck of the split; enough cases in the training set and enough of them in the test set. By keeping a very close gap in this situation I am running the risk of underfitting my model, is that correct?

If my way of thinking is not flawed, would a good solution be to allow for slightly higher metrics in the training set?

I have two questions (they can be solved by one answer). They are related to comparing the training metrics vs the validation metrics for each iteration / levels of regularization parameters. I am sorry for posting a long message but I am trying to explain my thoughts as clear as I can.

Generally from what I know we want to stop training when the validation metrics reach their max values (thus the so-called callback functions that are built-in in so many algorithms).

Question number 1: What happens if the validation metrics plateau? Should I relax my regularization parameters a bit because I am underfitting? (as to allow for a U shape in the validation metrics graph? U for logloss inverse U for AUC etc.) What if it still is stuck in the same value?. In that case what happens if I keep training without reaching an outrageous gap? This is closely related to my second question:

Question number 2: We generally want our training and validation metrics to hold relatively the same values, is that correct? However, what happens if for some reason the validation set and the training set are not representive of each other 100%? More specifically can there be some cases where the model is trained correctly (thus higher training metrics) and not overfit (i am not talking about huge gaps), while these cases are missing in the validation set (thus the validation metrics plateau)?I am asking because In real world, sometimes it is hard to gather huge amounts of data, thus some cases might not be well-represented. So even by performing k-fold validation n times, I may still be depending on the luck of the split; enough cases in the training set and enough of them in the test set. By keeping a very close gap in this situation I am running the risk of underfitting my model, is that correct?

If my way of thinking is not flawed, would a good solution be to allow for slightly higher metrics in the training set?

PS. I do not want to touch the test-set yet.