3
$\begingroup$

I'm currently working on a data set with two sets of samples. The csv file of the data could be found here. I would like to use KS test to see if these two sets of samples are from different distributions.

I ran the following R script:

# read data from the file
> data = read.csv('data.csv')
> ks.test(data[[1]], data[[2]])
    Two-sample Kolmogorov-Smirnov test

data:  data[[1]] and data[[2]]
D = 0.025, p-value = 0.9132
alternative hypothesis: two-sided

The KS test shows that these two samples are very similar. (In fact, they should come from same distribution.)

However, due to some reasons, instead of the raw values, the actual data that I will get will be normalized (zero mean, unit variance). So I tried to normalize the raw data I have and run the KS test again:

> ks.test(scale(data[[1]]), scale(data[[2]]))
    Two-sample Kolmogorov-Smirnov test

data:  scale(data[[1]]) and scale(data[[2]])
D = 0.3273, p-value < 2.2e-16
alternative hypothesis: two-sided

The p-value becomes almost zero after normalization indicating these two samples are significantly different (from different distributions).

My question is: How the normalization could make two similar samples becomes different from each other? I can see that if two samples are different, then normalization could make them similar. However, if two sets of data are similar, then intuitively, applying same operation onto them should make them still similar, at least not different from each other too much.

I did some further analysis about the data. I also tried to normalize the data into [0,1] range (using the formula (x-min(x))/(max(x)-min(x))), but same thing happened. At first, I thought it might be outliers caused this problem (I can see that an outlier may cause this problem if I normalize the data into [0,1] range.) I deleted all data whose abs value is larger than 4 standard deviation. But it still didn't help. Plus, I even plotted the eCDFs, they look the same even after normalization. Anything wrong with my usage of the R function?

Since the data contains ties, I also tried ks.boot, but I got the same result.

Could anyone help me to explain why it happened? Also, any suggestion about the hypothesis testing on normalized data? (The data I have right now is simulated data. In real world, I cannot get raw data, but only normalized one.)

Improve this question
$\endgroup$
13
  • $\begingroup$ Are the data paired? $\endgroup$
    – Glen_b
    Jan 12, 2015 at 3:14
  • $\begingroup$ Indeed, note that the correlation between values is quite strong. What is it you're actually interested in finding out? $\endgroup$
    – Glen_b
    Jan 12, 2015 at 3:33
  • $\begingroup$ @Glen_b No, they are not paired. I would like to see if these two samples are from same distribution. I think ks test could serve this purpose (partially?) To my understanding, ks test could identify significant difference between two samples. $\endgroup$
    – monnand
    Jan 12, 2015 at 3:44
  • 2
    $\begingroup$ I don't find this denial plausible, absent some other explanation. Note that large values ($>2\times 10^7$) only occur together. If the data are independent, how is this possible? There are 951 pairs of values (out of of 1000) that are both ($<2\times 10^7$) and 49 pairs of values that are both ($>2\times 10^7$), and no data whatever where one value is above that threshold and one is below. If these data were independent this would be astronomically unlikely. How do you account for clear dependence if they're not dependent? $\endgroup$
    – Glen_b
    Jan 12, 2015 at 4:16
  • 1
    $\begingroup$ Are the first transactions, second transactions, etc the same each time? [I don't think the effect I observed can plausibly be explained by chance. Your values have a correlation > 0.9. The chance of observing a correlation of that size with those margins if the data were independent is (again) astronomically small. Independence is not a plausible claim and so the KS test cannot apply. {There are numerous other issues in your post to deal with, but until we find out why your data are actually paired, the rest must wait.} ] $\endgroup$
    – Glen_b
    Jan 12, 2015 at 6:14

1 Answer 1

Reset to default
2
$\begingroup$

I hope to replace this with a full answer once we have sorted out what's going on.

In trying to show you what's going on with the dependence in large vs small values, I see another problem:

enter image description here

The y-values have been monotonically transformed (log(y-2.e6)) for clarity.

The green box shows how the large values always occur together.

But they're also astonishingly regular. It looks like the large ones are every 20th value.

The red ovals show another problem. Notice patches of almost pure-black followed by almost pure-blue in the red ovals? There's something weird going on. Why would the middling-size values alternate, with a patch from column 1 then a patch from column 2?

You have neither independence within columns nor within pairs, but I am not sure what is causing that particular alternation in the red ovals.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I just ran several other rounds of testing and collected another 8 sets of data (1000 points each). About the green box: Yes, in every set of samples, I could observe that the value will increase every ~20 samples. I think I'm pretty sure it is caused by the flushing mechanism in the database. (recap: The values are response times of a database write-query.) About the red ovals, I'm currently do not have a good explanation. However, does the column-independence really matter? I mean, what if I do a permutation for the data, which does not change the eCDF of the sample. $\endgroup$
    – monnand
    Jan 12, 2015 at 7:17
  • $\begingroup$ It's not just within-column-dependence there. There's a negative dependence in the medium-sized values (roughly in the range of $y$ where the red ovals are) -- when the the black ones tend to get that high for a period, the blue ones don't, and vice versa. It's a very complex situation, not merely positive dependence overall, but negative dependence in that region of values. Very odd. $\endgroup$
    – Glen_b
    Jan 12, 2015 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.