I am trying to find sigmoid function alternatives for logistic regression. I am curious that if I can replace sigmoid function by any cumulative distribution function, and what will be the best?
Other distributions may be less obvious, but still problematic. Consider the CDF of a Gamma distribution with $k=.5,\ \theta=1$ (the lavender line at the far left; figure copied from Wikipedia):
At a minimum, you are going to need a distribution whose support is $(-\infty, \infty)$ before you could consider using its CDF as a link function.
There are many (I suppose infinite) possible link functions that can be used, though. You don't have to use the logit, and it isn't necessarily the best (although we need to be more precise about what "best" means). You may be interested in reading my answers here: Difference between logit and probit models, or here: Is the logit function always the best for regression modeling of binary data?
I believe tanh(z) is a good replacement for the sigmoid function. Behaves almost exactly the same way. Also, the gradient descent update expression is the same