Given an arbitrary dataset, how would one decide whether to use GDA or Logistic Regression? Is the only way to choose via trying both and selecting the one with better performance or is there some way of determining which model to use pre-training?
Which analysis is more appropriate depends on the goal of the analysis, not only on the dataset. So what is your goal? In most cases with a binary response variable an estimate of risk (probability) will be of interest, so then logistic regression is indicated. See Why isn't Logistic Regression called Logistic Classification? why this is different from classification.
Even if the ultimate use is classification, first obtaining probability estimates is useful, since then an ultimate decision can be takes which uses loss functions which can vary from subject to subject, with unit characteristics. See for instance this answer by @Frank Harrell.