You are right that you are testing much broader assumption that the group labels are "random" and play no role in your results. As a proxy to test such hypothesis you use some test statistic that is evaluated on the permuted samples. In this case is median, but it could something else as well. So you test if the labels are exchangeable as tested using median as criterion to assess that.
|1 of 2|