You are right that you are testing much broader assumption that the  group labels are "random" and play no role in your results. As a proxy to test such hypothesis you use some test statistic that is evaluated on the permuted samples. In this case it is median, but it could something else as well. So you test if the labels are exchangeable as tested *using median as criterion* to assess that. In the end you will learn how "likely" would it be to find different medians in both groups if the group labels played no role in your data.