how do normality check in ks test assess for equivalence or difference in data sets?

Question

I have a series of data of photon counts versus time. These data are periodic, then I can fold them and obtain an average profile of the data. Nonetheless, some variations appear sometime in the single profiles. To check if the average profile and the single ones are statistically different, they suggested me to use the Kolmogorov-Smirnov test. I am new to this, but it seems very fascinating!

However, the suggested procedure is based not directly on the two data set (the averaged one and the singles ones), but on the residuals. Basically, I should subtract the single profiles to the averaged one, and check if the residuals are normally distributed.

What I do NOT understand is: how could the normality test on the residuals assess the equivalence or the difference of the two data sets?

Answer 1

how could the normality test on the residuals assess the equivalence or the difference of the two data sets?

It's essentially useless at that task (actually I feel my second and fourth comments already conveyed that).

Asking this is like asking to asking me to advise you on the best way to clean windows with a hammer.

A goodness of fit test is pretty much useless at the task, because its main functionality is almost orthogonal to the problem - it picks up the distribution of deviations, when what matters more is their typical magnitude, and perhaps direction.

As a thought experiment, consider

a) what happens with two fits, the second of which has every residual exactly 100 times the size of the corresponding residual of the first. A test of normality will give identical p-values, but they're not equally adequate. A normality test is precisely orthogonal to the direction of that aspect of the problem.

b) now instead imagine we have data that has a slightly heavy tailed distribution around its population curve - perhaps logistic errors, say, but you have exactly the correct functional form. With enough data, you'll reject normality, but that rejection tells you only that you might want to use something other than least squares to fit the model, it doesn't tell you you missed a bump on the curve.

As a result, the p-value for a straight goodness of fit test on the residuals tells you almost nothing about the actual issue of whether one curve matches a particular subset of data.

(By comparison, the discussion about the choice of which goodness of fit test to use is more akin to asking what kind of hammer might be best. If we had needed to hammer a Gaussian nail, the KS was almost entirely the wrong kind of hammer, but here we need some tool that's altogether different in its features and direction of operation.)