0
$\begingroup$

I need to compare performance of a web-application before and after some code change. We have several pages (call them $P_1$ and $P_2$; in practice there are more than 2), and for each we compare loading times. Let's say samples of $P_i$ loading time before the change are $s_{i1},s_{i2},\ldots$ and after the change $s'_{i1},s'_{i2},\ldots$. Currently we apply the Mann–Whitney U test to these samples separately for each $i$. However, I noticed that quite often the metric changes for all pages in the same direction, but is considered to be insignificant.

Is it a bad idea to take the union of the populations, so the test is applied to $s_{11},s_{21},s_{12},s_{22},\ldots$ and $s'_{11},s'_{21},s'_{12},s'_{22},\ldots$? At least I think this could make some of the cases described above significant, not be too likely to have false positive if (say) the change actually makes half the pages faster and the other half slower, and doesn't seem to violate the test's assumptions (as listed e.g. at https://statistics.laerd.com/statistical-guides/mann-whitney-u-test-assumptions.php).

Even if the above is a good idea, is there another, even better way?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

There are two reasons against your idea.

  1. Running tests that are chosen conditionally on the data to be tested are invalid. This means that if you see insignificant results and because of this you run a new test on the same data, any significance you then get will not be valid. This problem however goes away if you apply the new test on new data other than those based on which you got this idea, which I can imagine is possible in your situation.

  2. Chances are that the different pages are systematically different regarding their loading times. This means that taking the union of the populations will invalidate the i.i.d. assumption, as I'd expect different distributions for different pages.

If you are willing to make a normality assumption, a mixed model with a dummy variable for before/after and a random effect may do the trick. I'm not aware of a combination of Mann-Whitney with a random effect. A "quick and dirty" method would be to put samples together after subtracting an overall pagewise median, but this is nonstandard and may easily be criticised. (I'd probably do it for curiosity's sake but I'd be very careful not to overinterpret it, and would for sure check whether the different pages look similarly distributed after this operation.)

Improve this answer
$\endgroup$
5
  • $\begingroup$ 1. Yes, it would only be applied on new data (and on all new data). 2. They definitely are different. I didn't see the requirement for identical distributions there, but I was thinking that the resulting population could be considered as a sample from a multimodal distribution where we take a random page first and then a measurement from that page. But this doesn't really work, you are correct. Maybe it could be rescued by actually doing "take a random page and take one of its samples at random", but probably not. Thank you! $\endgroup$
    – Alexey Romanov
    Commented May 5, 2023 at 17:26
  • $\begingroup$ @AlexeyRomanov Technically the idea with the multimodal distribution is OK, however I'd suspect that because of an unmodelled source of variation the resulting test would have a worse power than one could achieve using the information from what page what observation comes. $\endgroup$
    – Christian Hennig
    Commented May 5, 2023 at 17:54
  • $\begingroup$ I also wanted to mention (but it got removed in the editing), that I definitely can't make a normality assumption. Some cases look close to log-normal, though I haven't done tests for it, and some are already bimodal. $\endgroup$
    – Alexey Romanov
    Commented May 5, 2023 at 19:19
  • $\begingroup$ And I'd be quite happy to use a test which takes this information into account if I knew a suitable test. $\endgroup$
    – Alexey Romanov
    Commented May 5, 2023 at 19:57
  • $\begingroup$ @AlexeyRomanov Many methods that assume normality also work well for non-normal data unless there are specific issues such as gross outliers. So don't think data have to be precisely normal. Also what I suggested may work, i.e., running the Mann-Whitney test on data minus the page-wise median, although this is currently not fully thought through. One might also do something using a permutation test approach. Unfortunately chances are that this is complex enough that it can't be solved just by asking a stranger on the internet. You'd need to collaborate with somebody who has access to the data. $\endgroup$
    – Christian Hennig
    Commented May 5, 2023 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.