The main difference is that logistic regression can only separate linearly separable classes where as SVM (with the kernel trick) can find any arbitrarily shaped decision boundary. This means that SVM will usually do better separating your classes (at least on your training set) but is more prone to over-fitting.
Linear regression is also a simpler model with fewer hyper-parameters to tune (zero if you're not using regularization) making it easier to implement.
Unless you have very good intuitions about the separability of your data, I would suggest start by fitting a logistic regression and if it isn't giving you satisfactory class separability, then try an SVM.
On the other hand, if you find that SVM is over-fitting no matter how you tune the hyper-parameters, consider trying logistic regression.
One final point - logistic regression outputs a probability of being in the positive class (you still need to choose a threshold to make it a classifer), SVM just outputs the classes. SVM can give you probabilies via Platt scaling but this can be very slow.