There seems to be a confusion in the premise of the question. In the second paragraph it says, "we could just use median regression as the OLS substitute". Note that regressing the conditional median on X is (a form of) quantile regression.
If the error in the underlying data generating process is normally distributed (which can be assessed by checking if the residuals are normal), then the conditional mean equals the conditional median. Moreover, any quantile you may be interested in (e.g., the 95th percentile, or the 37th percentile), can be determined for a given point in the X dimension with standard OLS methods. The main appeal of quantile regression is that it is more robust than OLS. The downside is that if all assumptions are met, it will be less efficient (that is, you will need a larger sample size to achieve the same power / your estimates will be less precise).