Comparing distributions from the same sample

Question

The Background:

I have two different sets of measures from a group of people who completed various choice tasks: 1) Demographic characteristics of each person. 2) Eye-tracking measures (fixation time, pupil size etc.) of each person.

My goal is to detect whether two different logit models can have different prediction powers. My first logit model (M1) has demographic variables only and my second model (M2) has eye-tracking variables only. So they are non-nested models, but the measures they use come from the same group of people.

I tried to see how these models perform in terms of out-of-sample predictions. So, I run cross-validation (I am not sure about the name of the procedure, but the logic is the same). So, in each iteration I randomly pick 90% of my data ( i.e. I am defining a training sub-sample) and fit my models (M1 and M2 separately) in the training sub-sample , then I am using the obtained fits to predict the responses in the 10% of the data (i.e. in the test sub-sample). I repeat it 1000 times for M1 and M2 independently.

Question: I have two distribution for M1 and M2 from this CV exercise. I expect that distributions are dependent (related) because one is the outcome of a demographic variables based model and the other is the outcome of an eye -tracking measures based model for the same group of people. How do I test that these two distribution are different?

Answer 1

Well, before asking this question I conducted a paired Wilcoxon test, the reason I was not sure about the validity of my test was the very high value of the test statistics. For instance, in one case I got this result with R:

wilcox.test(M2, M1, paired = TRUE, alternative = "greater")

Wilcoxon signed rank test with continuity correction

data: M2 and M1 V = 441940, p-value < 2.2e-16 alternative hypothesis: true location shift is greater than 0

Now I see that it is the correct way, if I compare the distributions not the means.

For dependent paired distributions one should use the t-test for comparing means:

t.test(M1, M2, paired=TRUE, alt="less")

Paired t-test

data: M1 and M2 t = -26.664, df = 999, p-value < 2.2e-16 alternative hypothesis: true difference in means is less than 0 95 percent confidence interval: -Inf -0.04439734 sample estimates: mean of the differences -0.04731914

Conclusion:

If you want to compare just distributions that paired and dependent use wilcoxon test, but if you want to compare the means use t-test.

check out this discussion as well. https://stats.stackexchange.com/a/55819/156139