It seems that both these refer to cases where the regressed (dependent) variable can only take certain values, as opposed to a linear regression.
So what is the difference between probit and logistic regression?
I think both will produce very similar results, however, the difference is the assumption about the distribution of the errors $\epsilon$ in $y=w^Tx + \epsilon$ (standard normal distribution in the probit model, standard logit distribution in the logistic model). The non-thresholded output of the probit model will be the z-scores of a standard normal distribution, whereas the output of a logistic model can be interpreted as probabilities.