This really comes down to an issue of "dimensionality". Imagine that each of the five "difference" values is a vector (i.e., a line pointing positively or negatively with length equal to the difference value). Geometrically, the mean (of) difference value is obtained by putting these vectors end-to-end in a single dimension, to get the aggregate difference, and then dividing by five. Contrarily, the RMSE is a Euclidean distance obtained by putting these vectors end-to-end in five different dimensions.
In terms of which measure is more suitable, this depends on whether the five difference values are measuring the same characteristic, or five different characteristics. If they are measuring the same characteristic then it is probably sensible that they be aggregated over one dimension. If they are measuring different things then it is more sensible to aggregate them over different dimensions using the RMSE.