Should overfitting or underfitting be concerned during hyperparameters tuning I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt / Hyperas, but I think the question applies to all the ways of searching the optimal hyperparameters. 
The idea of these libraries is to choose the best hyperparameters according to the performance of the metrics on a validation set. In my case, it’s MAE / MSE. As a result, for each of the individual tests, it takes the lowest / highest MAE / MSE of the validation set.
If I understand right, it has no way to detect if the model overfits or underfits. It means, even if the test produced very low validation MAE / MSE and I now evaluate the model on a test set, it might not produce good results e.g. due to overfitting problem. It also means I could have missed a good hyperparameter set because the model at the beginning performed very well but then started to overfit/underfit because of too many epochs. 
Should overfitting or underfitting be concerned during hyperparameters tuning? 
Should I look for the lowest or highest MAE / MSE during epochs?
 A: We tune our models to decrease the chance of overfit or underfit, by measuring the performance over the validation set(s). Some use just one validation set, but if not costly, it's better to have k-fold validation. But of course, this mechanism doesn't explicitly detect overfit or underfit. For example, your best method's validation MSE and training MSE differences could be high, signalling the possibility of overfit. Especially in neural networks overfitting can be due to over-training, and to detect it you should look at your training/validation metrics at each epoch, as you said (and set some early-stop recipe). Specifically for Keras, use EarlyStopping, with parameters patience, min_delta for setting your stopping criteria.
Not referring specifically to your situation, sometimes the problem lies in the data. In ML, we assume that test set comes from the same distribution as the training set, such that training set is a good representative of future samples. If your test set is very different, or if the training set is not enough to represent the space, you can't solve your problem even with the best algorithm.
