# how do normality check in ks test assess for equivalence or difference in data sets?

I have a series of data of photon counts versus time. These data are periodic, then I can fold them and obtain an average profile of the data. Nonetheless, some variations appear sometime in the single profiles. To check if the average profile and the single ones are statistically different, they suggested me to use the Kolmogorov-Smirnov test. I am new to this, but it seems very fascinating!

However, the suggested procedure is based not directly on the two data set (the averaged one and the singles ones), but on the residuals. Basically, I should subtract the single profiles to the averaged one, and check if the residuals are normally distributed.

What I do NOT understand is: how could the normality test on the residuals assess the equivalence or the difference of the two data sets?

• I don't think this is an appropriate way to test what it sounds like you're interested in - for a couple of reasons. Rather than ask how to implement some solution to the problem, why not ask the assembled experts about your underlying problem? Are you looking to check whether the within-period profiles are changing over time? Apr 10 '14 at 2:42
• I don't think this is an appropriate way to test what it sounds like you're interested in - for a couple of reasons What are these reasons? Rather than ask how to implement some solution to the problem, why not ask the assembled experts about your underlying problem? Sorry, I can't understand what you are asking. Are you looking to check whether the within-period profiles are changing over time? Not exactly, I want to check if there are systematic variations in the profiles, even if these variations are smoothed out in the averaged profile. Apr 10 '14 at 2:56
• Let me start with the second item. What I mean to say is the approach suggested to you won't be useful as-is. Yet you have somewhere up in the hundreds of actual expert statisticians who might potentially give substantially better answers, so why not begin at the beginning? Apr 10 '14 at 4:15
• third item: what does 'systematic variations' mean in a way that is different from what I asked about? This is likely to be the crux of your statistical issue. What kinds of variations? Before you saw the data, what were you expecting? Apr 10 '14 at 4:17
• First item: (i) the KS is a very low power test of normality. (ii) you already know don't have normality, you have COUNT data, which besides being discrete and non-negative, won't have constant variation as the mean changes. Why test what you know, a priori, to be certainly false? (iii) there are much better ways to identify those kind of things (iv) hypothesis tests are likely not answering the underlying questions you probably want to ask here -- it strikes me as probably the wrong tool for the job Apr 10 '14 at 4:20

how could the normality test on the residuals assess the equivalence or the difference of the two data sets?

It's essentially useless at that task (actually I feel my second and fourth comments already conveyed that).

A goodness of fit test is pretty much useless at the task, because its main functionality is almost orthogonal to the problem - it picks up the distribution of deviations, when what matters more is their typical magnitude, and perhaps direction.

As a thought experiment, consider

a) what happens with two fits, the second of which has every residual exactly 100 times the size of the corresponding residual of the first. A test of normality will give identical p-values, but they're not equally adequate. A normality test is precisely orthogonal to the direction of that aspect of the problem.

b) now instead imagine we have data that has a slightly heavy tailed distribution around its population curve - perhaps logistic errors, say, but you have exactly the correct functional form. With enough data, you'll reject normality, but that rejection tells you only that you might want to use something other than least squares to fit the model, it doesn't tell you you missed a bump on the curve.

As a result, the p-value for a straight goodness of fit test on the residuals tells you almost nothing about the actual issue of whether one curve matches a particular subset of data.

(By comparison, the discussion about the choice of which goodness of fit test to use is more akin to asking what kind of hammer might be best. If we had needed to hammer a Gaussian nail, the KS was almost entirely the wrong kind of hammer, but here we need some tool that's altogether different in its features and direction of operation.)

• Thanks Glen. I am particularly impressed by your statement:...it picks up the distribution of deviations, when what matters more is their typical magnitude, and perhaps direction. What do you mean by this? What is intended in this case for "magnitude" and "direction"? Is this referred to the so-called "outliers"? Apr 11 '14 at 7:13
• 'direction' is simply direction on the real number line ... i.e. the sign. It's not about outliers, at least not directly; the central issue lies elsewhere. The shape of the distribution of the residuals isn't especially important in telling you about good fit. It's the size (and in some cases sign) of the residuals. Apr 11 '14 at 9:36
• Ok, I think I am starting to grab the point. I was now thinking to change the strategy and use the 2dataset KS test on the model and on the single profiles, avoiding to work through the residuals. Is it worthy to open a new topic about this? Apr 11 '14 at 9:42
• Probably, since it now seems to be a substantively different question. Apr 11 '14 at 10:19