Quantitatively, how powerful is Shapiro-Wilk or other distribution-fit tests for small sample sizes? I'm looking for an analysis (I assume a book or website, but feel free to put it all in comments here) that provides an in-depth discussion of the power/accuracy of normality assessments such as Shapiro-Wilk.
In short, I understand that Shapiro-Wilk "is good for small data sets" and with large data sets it is overly sensitive to small deviations from normality - but what does this mean quantitatively?
Overall I'd like to get a feel for something like this:
If I have n data points, I'm mm% confident that the Shapiro-Wilk (and/or other tests) test is providing me a reasonable conclusion.
How does mm% change when n is 15 ... 10 ... 7?  How about as it goes from 100 to 500 ... 1000?
Please note that I do believe in the power of graphical assessments as well, but I'm keeping those out of scope for this question and I'm looking for numeric, quantitative assessments in this case.
 A: What can be simulated is the power of the Shapiro-Wilk test for detecting specific non-normal distributions. This depends strongly on the exact distribution you want to detect. For this reason a plethora of results can be considered, and you can find examples for the test working well and not so well. (This means that data from a certain non-normal distribution with a certain sample size is simulated, and by this you can determine, at least approximately, the probability for the S-W test to reject normality; the higher the better.)
Here is some work that simulates the power of the Shapiro-Wilk test:
Hadi Alizadeh, N. R. Arghami (2011) Monte Carlo comparison of seven normality tests. Journal of Statistical Computation and Simulation 81(8):965-972
https://www.researchgate.net/publication/232942214_Monte_Carlo_comparison_of_seven_normality_tests
Hadi Alizadeh Noughabi (2018) A Comprehensive Study on Power of Tests for Normality. Journal of Statistical Theory and Applications 17. 647 - 660
https://www.atlantis-press.com/journals/jsta/125905578/view
Razali, N. M. & Wah, Y. B. (2011), ‘Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests’, Journal of Statistical Modeling and Analytics 2, 21–33.
https://www.nrc.gov/docs/ML1714/ML17143A100.pdf
Danush K. Wijekularathna, Ananda B. W. Manage & Stephen M. Scariano (2019) Power analysis of several normality tests: A Monte Carlo simulation study, Communications in Statistics - Simulation and Computation, DOI: 10.1080/03610918.2019.1658780
Some older work is cited in Thode, H. C. (2002) Testing for Normality. Marcel Dekker Inc.
A: You are asking for the probability of a "reasonable conclusion". You can get this if and only if you give a precise enough definition of a reasonable conclusion. I too would love a procedure that reliably tells me when deviation from normality is important enough to matter, but there is a gradation from sample sizes of 1 and 2   -- for which any sample whatsoever is totally consistent with a normal distribution -- to arbitrarily large sample sizes -- for which trivial deviations will return significance levels below conventional thresholds.
What is important enough to matter depends on your purposes and on your view of the data, as statistics is a craft drawing on personal expertise and experience just as much as it is codified technique. For example:

*

*Marginal normal distributions are only rarely required or even ideal.


*In practice if I found very close approximations to conditional normal distributions -- e.g. in residuals from models known to be sound scientifically for high quality data -- I would suspect fraud more frequently than I would believe the result implicitly.
It's quite hard to teach beginners how to sit loose to significance testing, not least in disciplines that persist in over-rating it, and because you have to look at many datasets before you start getting your own independent sense of what helps. But the existence proof that beginners can grow into experts is the fact that there are experts.
EDIT A general purpose "test" is to post a normal quantile plot of the data together with several normal quantile plots for random samples of the same size from a normal. Convenient numbers of simulated samples might be 24, 35, 48, ... permitting, when combined with the original, a 5 x 5, 6 x 6, 7 x 7, ... display. This is the line-up test, similar to the idea that a suspect should be shown to witnesses together with various arbitrary people. If the suspect isn't identifiable from the others, the case for being different isn't supported. Although quite often re-invented fairly recently, the idea is in Walter Shewhart's main book (and may yet be older). One attraction of the test is the Anna Karenina principle that each non-normal distribution may be non-normal in its own different way.
A: As you allude to when you bring up the use of graphical examination for this task of assessing normality, the answer is $0\%$ $\forall$ $n$.
