Yes, you are justified in being concerned. You are theoretically justified when the mean does not exist, as would be the case in stock market returns, in economic growth in certain cases, or in cancer research. You would also be justified if the estimator of the mean was less efficient than the estimator of the median for the problem you are facing and you formally solved for efficiency. It is also reasonably safe when you have no mechanism to determine the likelihood or density function you are working with and you believe robustness is important. There are circumstances where a mean can be very fragile.
Using the median has a different interpretation from using the mean. Theories that are based on expectations are having their assumptions violated by using the median in estimation. You are no longer testing the theory you are claiming to test if an expectation operator is present. Further, a mean implies a convergence and a direct relationship to the underlying derivative. The median does not.
For example, if y=5x+2, where the 5 was estimated using ordinary least squares, then it implies that if x=5 then y will converge to 27. For median regression, such as Theil's regression, it would imply that half the time $y\ge{27}$ and half the time $y\le{27}$. You cannot make an stronger interpretation than that.