Is there a way to quantify how appropriate a t-test is for given data? I'm doing some programming and analysis for a post-doc who used to do this workflow using Excel and StatPlus. Apparently StatPlus has some feature where you tell it to compare two samples, and it will automatically decide whether a t-test (with heteroscedasticity) is appropriate and then give you either that, or a suite of non-parametric tests (like Mann-Whitney). This feels like a Bad Idea™ to me, but even if I were to faithfully try to replace this feature in my R code, I wouldn't know how best to make the decision "to t, or not to t".
So, (a) is there a way to quantify how well you expect a t-test to perform for a given data set, even if there isn't some clear decision boundary beyond which you should use a different test instead? I know that the t-test theoretically doesn't work on non-normal data, but also that it can be quite robust to non-normality in some circumstances (and possibly better than the non-parametric alternatives). OR (b) can you help me explain to my collaborator why this is a bad idea, and what we should be doing instead?
 A: Some comments. I'm assuming that you mean the t-test for two independent samples, although most arguments also hold for the one-sample test (mostly in a straighter way).
(1) "I know that the t-test theoretically doesn't work on non-normal data." This is not true. The t-test is based on theory that requires normality, meaning that it does work (in a certain sense of the word) for normal data. No theory says that the t-test is bad for non-normal data in any generality (it is known to be bad with specific non-normal distributions, particularly where variances don't exist). In fact, in a number of non-normal situations the t-test can be theoretically justified on Central Limit Theorem grounds if the sample size is large enough.
(2) Generally the appropriateness of the t-test (as of any statistical method) cannot be assessed based on the data alone, but also depends on issues such as whether the data are randomly sampled, whether the hypotheses are appropriate for the research question of interest, whether there are hidden dependencies (that cannot always be diagnosed from the data), what is known about the reliability and validity of measurements etc.
(3) Note that the standard null hypothesis and alternative of the t-test are defined in terms of the normal assumption. These need to be adapted if the underlying distributions are non-normal. This may seem trivial at first (one may still say that the H0 is that means are equal and the alternative that they differ - as long as means exist which in general isn't guaranteed), however there's a catch. In a real situation it can easily happen that not only variances differ between the two groups, but also the distributional shapes. For normal distributions, the mean, median, and center of symmetry are the same. For general distributions this does not normally hold. Meaning that testing whether means are the same (against them being different) may be a different problem with a different result than testing whether medians are the same. Imagine for example that you have in one group a mixture distribution $0.99{\cal N}(0,1)+0.01{\cal N}(100,1)$ and in the other group ${\cal N}(1,1)$. These two have the same mean but are very different, and the median of the second group will be much higher than the median of the first. In order to check whether the t-test "works" you will need to decide first whether in such a situation you think that the "correct" result is "reject" (because the second group dominates 99% of the first group) or "not reject" (because the means are the same). This decision is a personal one, depending on the situation, and cannot be made by the data, nor is there a "mathematically correct" answer to this. It depends on this decision what it even means to say that "the t-test works".
(4) Some hardcore Bayesians may say that significance tests are never appropriate.
(5) Violations of the independence and identical within-group distribution assumption are often much worse than non-normality, so these need to be checked, too. Unfortunately, this is not always possible, for example if you don't know in which order the observations were made, or if dependence or non-identity is irregular so that plots and standard tests cannot pick it up. One reason more that it is essential to know how the data were obtained.
(6) Even if we're willing to assume i.i.d. within groups, using any automatic data-dependent rule in order to decide whether to run a t-test or, say, a Mann-Whitney test is problematic, because this in itself violates the assumptions (conditioning on the decision rule accepting normality, the distribution of the data is different from unconditionally, meaning that if the data were i.i.d. distributed normally before, they are no longer afterwards). A number of authors who have investigated the combination of a normality test and then t-test conditionally on not rejecting normality recommend against such a practice. See Example 1 and Sec. 5.3 for an overview in this paper:
https://arxiv.org/abs/1908.02218
I. Shamsudheen and C. Hennig: "Should we test the model assumptions before running a model-based test?"
These authors argue that preliminary normality testing may have its advantages and is somewhat too pessimistically portrayed in the literature. They don't recommend a fixed rule though.
(7) You may probably want to define whether the test "works" in terms of type I and type II error probabilities. Note that in any case the transition between satisfactory and unsatisfactory results is gradual and continuous, meaning that specifying a sharp cutoff between "works" and "does not work" will be somewhat arbitrary (even if there were no other issues).
A: 
(a) is there a way to quantify how well you expect a t-test to perform for a given data set, even if there isn't some clear decision boundary beyond which you should use a different test instead?

I would argue that if such a process were to exist, it would not be a quantitative one.  Appropriateness of a test really stems from context.  Ask yourself:

*

*What is my data generating process?

*Are the properties of my data compatible with the t-test?

*Is my sample size sufficiently large to justify use of the t test?  What evidence do I have for this?

What I imagine StatPlus is doing is some background tests (e.g. Shaprio Wilk) and then using some if/then logic to return to you whatever it returns.  This is a bad idea, if not only because large enough samples can return correct but useless results.  A student t distirbution with 20 degrees of freedom can be normal enough for a lot of applications, but its up to the user to decide if the differences between the t and the normal are large enough to warrant not using the former for the latter.  That isn't something a computer is going to know but it is something you could potentially reason about.
