In a neural network, how to assess the impact of different activation functions on overfitting? I would like to see how the type of non-linearity (sigmoid, tanh, ReLU) affect overfitting behavior of a neural network, and I am not sure what's the proper way to do it.
More specifically, should I also consider the convergence of each model in this case? Let's say I set the threshold for the error rate to be 0.1, and should I consider the overfitting behavior only if all the models will be able to converge at this threshold at a given set of parameters? What if different models converge at different iterations? And the corresponding overfitting may appear before or after the convergence point. What's the proper parameter to evaluate overfitting problem? For example, at which iteration the overfitting appears?
 A: If networks share the same structure and their activations have the same representational power—meaning that they can represent the same set of functions—you'd expect them to settle on the same optimal weights. But, that may not be known for every possible pair of compared activations. This leaves you with, as I see it, two questions:


*

*Are any activations more prone to overfitting?

*Do any overfit sooner?


Measuring overfitting
Recall that overfitting means, briefly, that your learning method has chosen some model over another model that meets two criteria:


*

*It outperforms the chosen model on held-out test data.

*It underperforms the chosen model on training data.


In the neural network case, one typically avoids overfitting by stopping training once test error begins to increase. (Any iteration thereafter will produce weights that overfit, at least in the case of standard backpropagation without any means of breaking out of local optima.) Most interest is usually place on how well the learner performs on held out data, thus why test error is often used to assess overfitting.
Once you've established each activation's relative performance, then you may wish to consider whether one reaches optimal test error sooner. (Again, equivalent to the point at which overfitting could begin to occur.) Iterations is a straightforward way to measure this. Though, different activations may have different computational burdens, and you may wish to account for this using wall-clock time.
Should we consider convergence?
You mention one important fact about convergence, that overfitting may occur before or after convergence to error within a chosen threshold. The first iteration within threshold may not be optimal: The best test error may be several more iterations away. 
Say you were working with the UCI handwritten digits dataset. I'm not sure of ANN benchmarks of this dataset offhand, but I recall achieving test error of $0.02-0.03$. (The classification was to determine whether a given digit was a 9.) If your comparison ceased at $0.1$, your comparison wouldn't be showcasing the best results available to each activation.
In the other case, even the best method may not converge within a certain error, and you'd likely still want to know whether and when this network began to overfit.
Thus, rather than structuring comparisons based on convergence to a predetermined error threshold, you may wish to observe each neural network until its test error has increased over some number of iterations.
A: 
In overfitting you are interested in error on your training and test set. If your training set error is dropping but your test set is increasing (as on the picture) then you are overfitting. So this is what you should be looking at. You can be looking at on which point ooverfitting appears, or how severe it can be. But number of neurons and regularization will have way higher impact than activation function
