TheA short answer is that this is contentious. Contrary to the advice you mention, people in many fields do take means of ordinal scales and are often happy that means do what they want. Grade-point averages or the equivalent in many educational systems are one example.
A stronger reason for not using the mean with ordinal data is that its value depends on conventions on coding. Numerical codes such as 1, 2, 3, 4 are usually just chosen for simplicity or convenience, but in principle they could equally well be 1, 23, 456, 7890 as far as corresponding to a defined order as concerned. Taking the mean in either case would involving taking those conventions literally (namely, as if the numbers were not arbitrary, but justifiable), and there are no rigorous grounds for doing that. You need an interval scale in which equal differences between values can be taken literally to justify taking means. That I take to be the main argument, but as already indicated people often ignore it and deliberately, because they find means useful, whatever measurement theorists say.
But often also people are asked (say) on a scale of 1 to 5, how do you rate something?
Examples abound: websites, sports, other kinds of competitions and indeed education. Here people are being shown a scale and being asked to use it.
It is widely understood that non-integers make sense, but you are just being allowed to use integers as a convention. Is this ordinal scale? Some say yes, some say no. Otherwise put, part of the problem is that what is ordinal scale is itself a fuzzy or debated area.
Consider again grades for academic work, say E to A. Often such grades are also treated numerically, say as 1 to 5, and routinely people calculate averages for students, courses, schools, etc. and do further analyses of such data. While it remains true that any mapping to numeric scores is arbitrary but acceptable so long as it preserves order, nevertheless in practice people assigning and receiving the grades know that scores have numeric equivalents and know that grades will be averaged.
One pragmatic reason for using means is that medians and modes are often poor summaries of the information in the data. Suppose you have a scale running from strongly disagree to strongly agree and for convenience code those points 1 to 5. Now imagine one sample coded 1, 1, 2, 2, 2 and another 1, 2, 2, 4, 5. Now raise your hands if you think that median and mode are the only justifiable summaries because it's an ordinal scale. Now raise your hands if you find the mean useful too, regardless of whether sums are well defined, etc.
Naturally, the mean would be a hypersensitive summary if the codes were the squares or cubes of 1 to 5, say, and that might not be what you want. (If your aim is to identify high-fliers quickly it might be exactly what you want!) But that's precisely why conventional coding with successive integer codes is a practical choice, because it often works quite well in practice. That is not an argument which carries any weight with measurement theorists, nor should it, but data analysts should be interested in producing information-rich summaries.
I agree with anyone who says: use the entire distribution of grade frequencies, but that is not the point at issue.
