Why are measures of dispersion calculated relative to some central point? Why wouldn't, for instance, all possible non-repeated, pairwise differences in the dataset be a valid measure of spread?
Actually, not all measures of dispersion are calculated relative to some central point. Examples include the $Q_n$ and $S_n$ statistics. Your intuition is sharp, as indeed, the calculation of these only relies on pairwise differences.
A downside of these two estimators is that they are less efficient at the gaussian distribution than the classical variance. However, an advantage is that they are more robust.