There are too many parameters while building an artificial neural network. Some of which that comes to my mind are:

  1. Number of layers
  2. Types of layers
  3. Number of nodes in each level of layer
  4. Activation functions in each layer
  5. Ordering of the layers
  6. Different types of optimizers
  7. Different types of loss functions
  8. Batch size while fitting
  9. Epochs while fitting

The lone fact that the number of layers might be in the order of tens and the number of nodes might be in the order of thousands implies that we can end up with billions of different combinations. Considering all the parameters in the list above, how can one know that their model is close to the optimum ?

The cases I study on does not require me to be too precise in terms of r2_score. In other words, it would not make a huge difference if the r2_score is 0.85 or 0.87. But the question is how do I know if I am real close to the saturation point of the r2_score for the given data, i.e. the optimum model would yield an r2_score of 0.90 and I am not stuck at 0.80?