gunes has already presented a wonderful answer with simple formulas. Here is my a numerical example to test it: consider the set {1, 1, 1, 1, 1, 1, 1, 1, 1, 11}; that is, nine 1s and a single 11. The mean is 2, the median is 1.
When you consider the sum of absolute values as the sum of distances, the median will have 0 distance to nine values but a distance of 10 to the final value, making a total sum of 10. By moving our comparison value a single unit to 2 (the mean), we will increase the sum of distances by 1 from each of the nine values and decrease it by 1 from the single final value, making a total sum of 17.
Applying the formulas from gunes, when you move right from the median by any small value of $\Delta$, you add $9 * \Delta$ and subtract $1 * \Delta$, which means the sum increases by a total of $8 * \Delta$. If you move left then the sum increases by a total of $10 * \Delta$.