0
$\begingroup$

I would like to compare task time to completion distributions (measured in minutes) across two groups. The distributions are highly skewed (the means are 300 to 400 times larger than the medians) and multimodal. I have a few thousand observations per group.

In particular, I am interested in testing whether the test group is completing the task faster.

Examining the distributions visually and comparing their quantiles, the test group appears to show faster task completion times.

I've tried using the Whitney Mann U test and Kolmogorov Smirnov test (like below) testing for whether the test group time to completions are faster, conducting one-sided tests. The KS test gives a smaller p-value.

Which test is more appropriate in this use case? I've read that the KS test has more power when testing continuous variables. Is this correct? And would that explain why it's giving smaller p-values?

wilcox.test(test, control, alternative = "l")
ks.test(test, control, alternative = "g")
Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Impossible to say just from the information provided. Multi-modality and skewness could have various important impacts on how the two tests perform. What do you mean by one group finishing faster? On average? Fastest are faster? Fewer take longer than specific benchmark? // How many in each group? Enough that the answer is just obvious looking at histograms? $\endgroup$
    – BruceET
    Commented Feb 7, 2021 at 4:55
  • 1
    $\begingroup$ I have a lot of trouble believing that KS is remotely appropriate for your task. KS will be sensitive to much more than a shift in mean or median. It will, for instance, catch that $N(0,1)$ and $N(0,7)$ are different, but it will not give insight into the way that they are different; those distributions do not differ in the way that seems like it would be most important to you. // I am with Bruce that you should refine what you mean about one group being faster. $\endgroup$
    – Dave
    Commented Feb 7, 2021 at 5:55
  • $\begingroup$ With your sample sizes, detailed descriptive statistics could be revealing. Try relative distribution methods, see stats.stackexchange.com/questions/28431/… Share some visualizations with us. Can you share (a link to) your data, or some mockup? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Feb 7, 2021 at 15:14
  • 1
    $\begingroup$ Mann Whitney test provides an answer to the question “what is the probability that a (randomly selected) person in the test group completes the task faster than a person in the control group?” That sounds to me like the answer to your question. But, I agree with the other comments that you should decide more clearly what question you want to answer. $\endgroup$
    – John L
    Commented Feb 7, 2021 at 18:31
  • $\begingroup$ Ideally, I would like to test whether the times to complete the task distribution shifted to become faster, across the entire distribution (not just the median or a given quantile). $\endgroup$
    – Harry M
    Commented Feb 7, 2021 at 23:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.