Are there any good tests that can be used for an arbitrary distribution against a generalized alternative (perhaps subject to some regularity conditions for it to be reliable)?
There are - but not when you have to estimate parameters.
For many distributions you could do something akin to a Shapiro-Francia test, but the distribution of critical values would be different for each distribution.
You could also do something like a Lilliefors test ... and again, critical values would be different for each distribution.
For example, if I think my distribution is gamma or log-normal, are there any well-understood tests, with a literature I could find describing when they work, that would reject those choices if the data were clearly bi-modal, or strongly fat-tailed or thin-tailed relative to what is expected given a range of parameter values that is broadly consistent with the data?
In the case of the lognormal, take logs and test for normality using a test suitable for testing normality with estimated parameters, such as the Shapiro-Wilk.
In the case of a gamma, you could:
perform a Lilliefors-type test (a Kolmogorov-Smirnov-type test where parameters are estimated, but the distribution is simulated to take account of the effect of the esitmation)
do the same type of thing with an Anderson-Darling test (this tends to be much less affected by parameter estimation - see the discussion in D'Agostino & Stephens Goodness of fit Techniques.
test the correlation in a gamma probability plot (that's what I mean by a test akin to the Shapiro-Francia). For example, for an approximate test, one could perform a Wilson-Hilferty type approach (i.e. a cube root transformation) and test normality, though there are other approaches for "gamma-probability" type plots.
There's also a smooth test for the gamma that should be very suitable:
De Boeck, B., O. Thas, J. C.W. Rayner & D. J. Best, (2011)
Smooth tests for the gamma distribution
Journal of Statistical Computation and Simulation
Volume 81, Issue 7, p843-855
Since many people haven't heard of these, I'll give a quick outline of them: Smooth tests consist basically of parameterizing the log-density of the data as the hypothesized log-density plus a set of orthonormal functions (that is, orthogonal to the null as well as each other); the functions differ for each density tested. There are some packages that do smooth tests in R, but I don't know that the gamma one is in there at the moment. They do deal with parameter estimation in a very simple way and generally have good power properties.
R packages that do smooth tests include ddst and smoothtest.
--
edit to respond to questions in comments:
where I could hand my data and my preferred distribution to a package and say"Is this a __ distribution?" and get back some result that would let me answer this question in a straightforward way
You really can't get a "*Is this a __ distribution?*" test. Imagine for a second that there was something like a K-S test but where you could estimate parameters (i.e. a truly omnibus test). Even so, in general for a given sample there would be an infinite number of distributions which would not be rejected by such a test. Statistical tests can help you rule out things ('that's clearly not exponential!') but not to say "that must be Weibull" or something like that, in the same way that you can say "the data are plainly inconsistent with a population mean of 100" but not to say "well, the population mean must be 157.32". Failure to reject a null doesn't tell you the null is true.
Instead, I would need to construct an analogue to one of the normality tests, and then compute my own critical values by Monte Carlo methods or something. Is that right?
Pretty much.
Well, there's the chi-square tests, I suppose.
However, I think you'd probably be best served by pursuing the Anderson-Darling test, but adjusting for parameter estimation.
I think potentially it comes at least fairly close to doing what you want (again, see D'Agostino and Stephens' book on that). In particular, it may be that fairly simple kinds of adjustment to critical values to deal with parameter estimation may be possible in that case.
