By "the skewness" I guess you mean the moment-based skewness calculated from third and second powers of deviations from the mean. But there are other ways of thinking about skewness that perhaps come closer to what you seek. You can measure skewness by a measure using mean, median and standard deviation (SD):
(mean $-$ median) / SD
Like any other similar measure, that ratio is a reduction that allows different shapes to be represented by the same value (and, notably, is necessarily zero when the mean and median are equal, regardless of whether the distribution is symmetric). So the distance between mean and median is pertinent to skewness and is made a measure of skewness (so unitless) by dividing by the SD. The measure is also interesting because it must fall in the interval $[-1, 1]$. This measure can be traced to Karl Pearson, although he did not do much with it.
Other measures start with
[(upper quartile $-$ median) $-$ (median $-$ lower quartile)] /
(upper quartile $-$ lower quartile)
which also falls in the same interval $[-1, 1]$. The same idea can be applied to any other symmetrically placed quantiles. As with the first measure, the numerator has the same units as the variable, and is made a measure of skewness by dividing by a reference distance. This measure has been attributed to Francis Galton, who did not propose it, although he did discuss measures in similar spirit; to Arthur Lyon Bowley, who did evidently invent it; and to George Udny Yule, who might have invented it independently but published after Bowley.
That's far from the end of what has been proposed or is possible, but I will stop there.