gunes has already presented a wonderful answer with simple formulas. Here is my a numerical example to test it: consider the set {1, 1, 1, 1, 1, 1, 1, 1, 1, 11}; that is, nine 1s and a single 11. The mean is 2, the median is 1.
When you consider the sum of absolute values as the sum of distances, the median will have 0 distance to nine values but a distance of 10 to the final value, making a total sum of 10. By moving our comparison value a single unit to 2 (the mean), we will increase the sum of distances by 1 from each of the nine values and decrease it by 1 from the single final value, making a total sum of 17.
Applying the formulas from gunes, when you move right from the median by any small value of $\Delta$, you add $9 * \Delta$ and subtract $1 * \Delta$, which means the sum increases by a total of $8 * \Delta$. If you move left then the sum increases by a total of $10 * \Delta$.
By intuition, you stop moving left or right when the number of values to the left and to the right are equal. This can be the median (middle value) for an odd number of values or anywhere between the two middle values when there are an even number of values.
To demonstrate the final point: consider the set {1, 2, 3, 4}. Here the median is 2.5 by definition. But you can use an point between the two middle values (inclusive) of 2 and 3. The sum of distances would be 4 for the [2, 3] range (for 2, 2.5, 3, or anything in between).