Under what conditions do different choices of link function for GLMs result in considerably different models? The statistical folklore I have heard is that the choice of link function usually does not considerably affect the resulting fit of a GLM. For example, usually probit regression and logistic regression give models that generate similar predicted probabilities. D. R. Cox made a similar comment in Analysis of Binary Data.
How would one mathematically describe this similarity and under what conditions it holds?
 A: Both the Probit (normal distribution Link Function) and the Logit (Logistic distribution Link Function) are symmetrical and are not too different from each other. A good explanation (with History) can be found J.S. Cramer (2002), The Origins of Logistic Regression  https://www.econstor.eu/bitstream/10419/86100/1/02119.pdf
The non-symmetrical such as Gompit (Gompertz Ln(-Ln(1-p)) or it's negative) link functions differ more than the symmetrical link functions.
Here is a graphical comparison between the Normal CDF (Probit) vs the Logistic CDF scaled to standard deviation (using Sqrt(3)/π). Note they are very similar. The Logistic has heavier tails, and higher probability of being near the mean as can be seen by the gap ~ 1σ (ref excess Kurtosis).
From a mathematical standpoint they are highly correlated (in Plot below: R=0.997, R-squared=99.4%). Here is a linear plot for a sample size of 250 using median ranks to estimate (p) (~MR=(rank-0.3)/(n+0.4)). Note they primarily differ beyond +/- 2 Standard Deviations (σ). In a sample size of 250 that represents ~ 5 individual samples beyond 2σ in each tail.  I hope this helps.

