If the parent distributions are normal, then the $t$-statistic
$$t = \frac{\text{Mean}_1 - \text{Mean}_2}{\sqrt{\dfrac{s_1^2}{N_1}+\dfrac{s_2^2}{N_2}}}$$
has the same distribution as
$$u = \frac{\text{Median}_1 - \text{Median}_2}{\sqrt{\pi/2}\,\sqrt{J_1+J_2}}$$
where
$$J_1=\frac{IQR_1^2}{1.82 N_1}, \ \ J_2 = \frac{IQR_2^2}{1.82 N_2}$$
This is because the variances of sample medians are $\pi/2$ times the variances of sample means, and the IQRs are $\sqrt{1.82^{\phantom'}}$ times the standard deviations.
So if the populations are normal or roughly normal, you can apply the $t$-test to $u$, using
$$\nu=\frac{(J_1+J_2)^2}{\dfrac{J_1^2}{N_1-1} + \dfrac{J_2^2}{N_2 -1}}$$
degrees of freedom.
This follows Welch's $t$-test, and when the variances or sample sizes are equal, the formulas can also be simplified to follow Student’s $t$-test.