Consequences of violating assumption of equal variance Mann-Whitney U test for location shift model

I've read through several posts on the WMN test, but can't seem to find information on this question. What are the consequences if one assumes equal scale in distributions for a location shift model using the Mann-Whitney U test? I would like to test equality of medians between two groups (and calculate a confidence interval for the shift in median), but due to extreme values the variances tend to be quite different. In reality, I could assume the population distributions have equal variances, but I am wondering what the consequences are for my conclusions in the median difference estimate and the confidence interval if the assumption is not true in reality.

Here are some results for what I'm looking at. If I remove extreme values from Group 1, the variances do become more similar, but this requires removing around 10% of the data from Group 1.

Group 1:

count        373.00
mean       19413.72
std       279752.91
min      -291124.25
25%           39.80
50%           53.27
75%           69.97
max      4264238.32


Group 2:

count    23.000000
mean     54.952899
std      16.609280
min       9.416667
25%      44.283333
50%      55.900000
75%      66.733333
max      80.850000


Group 1 with extreme values removed:

count    329.000000
mean      50.527102
std       22.833582
min        0.000000
25%       39.666667
50%       52.600000
75%       64.350000
max      113.650000