As the title says, what is exactly what is being tested before deciding to use a non-parametric alternative test (as Kruskal-Wallis for ANOVA, or Mann-Whitney's U for student's t)?
Most sources are not specific on this issue, being content with just referring to the "normality of the data" (e.g. statistics how to). This leaves us with two options, either the normality of the residuals or the normality of the response variable, and looking online both arguments are used.
For "data normality" as "response normality" we have other sources (e.g. Boston University School of Public Health, and here). For example, in the Biostatistics Handbook by John McDonald which on his Kruskal-Wallis section refers to the normality assumption, which gives a sequence of examples of normality testing on histograms and distributions of different response variables:
Parametric tests assume that your data fit the normal distribution. If your measurement variable is not normally distributed, you may be increasing your chance of a false positive result if you analyze the data with a test that assumes normality.
Also stating that residuals come as a second option:
If there are not enough observations in each group to check normality, you may want to examine the residuals (each observation minus the mean of its group).
On the other hand, for "residual normality", Wikipedia is on board with it:
Since it is a non-parametric method, the Kruskal–Wallis test does not assume a normal distribution of the residuals, unlike the analogous one-way analysis of variance.
As well as other answers on this website (where in fact the accepted, but not top voted, answer refers to response variable normality)
I am inclined to think it is residual normality what matters, but I am actually curious on why it seems to be such a common error—that actually makes its way into academic contexts, handbooks and "general knowledge"—and if there's some weight or sense into the "response variable normality" position.