2
$\begingroup$

I am doing a statistical test (analysis) for the following case: As part of a product aimed at improving the quality and speed of code writing for developers, we have implemented a new feature that should make code review faster.

  • Checking the time for code reviews in these groups :
  1. Before the implementation date of the feature
  2. After the implementation date of the feature
  • Each group represents a vector of times in minutes
  • The 2 samples distribution is unKnown (abnormal)
    I have tried several transformations to bring about a normal distribution, but without success. So I used U-test
  • I use Python to write My question is:

What statistical significance test should I use when I have an abnormal distribution with time data and not numerical data?

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to Cross Validated! What do you mean by time data instead of numerical data? Do you mean that your Python code stores a datetime object instead of a raw number? $\endgroup$
    – Dave
    Commented Apr 25, 2022 at 9:26
  • $\begingroup$ Specifically, I measure time data for both groups: the duration of an action (in minutes). * The data type is less important What statistical significance test should I use? This is the focus of the question. Thanks $\endgroup$
    – lincoln65
    Commented Apr 25, 2022 at 11:49
  • $\begingroup$ You mention the Mann–Whitney–Wilcoxon rank sum test (U-test). It should work well for your data, as it doesn't assume the variables are normally distributed. So you have (at least) two options: U-test and permutation test. As an side: duration (measured in minutes, hours, etc) is a numeric variable, though obviously non-normal as duration is nonnegative and very often right skewed. $\endgroup$
    – dipetkov
    Commented Apr 26, 2022 at 9:44

1 Answer 1

Reset to default
4
$\begingroup$

For testing a difference between the means of both groups, a t-test (aka "Welch test" to make clear that no assumption of equal variances is made) can even be used when the data of both groups are not normally distributed: for a sufficient number of samples, the mean value is approximately normally distributed due to the Central Limit Theorem.

A non-parametric alternative is a permutation test: for a number (say $K=10000$) of times, permute the group labels in the data and compute the difference between the group means. The p-value is the proportion of permutations for which the difference is greater than the actually observed difference.

The usual caveat applies: statistical significance does not mean importance, and you should therefore also check whether the difference is large enough to be of practical relevance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.