To give a counter-example, you almost certainly want the mean and not the median when calculating returns in finance.
Examples where the median is horribly misleading compared to the mean:
- If you're looking at a set of bond returns, the median will effectively ignore those observations where the bond defaults and you lose half or more of your money.
- If you're looking at venture capital returns, it's in some sense the reverse. The median company in VC or angel investing is a bust, and the median will effectively ignore the big winners like Google. The return for Ron Conway's first angel fund came largely from one company, Google.
Sometimes insensitivity to outliers is NOT what you want!
Good luck explaining to investors, "I know our fund is down 40 percent this year because nearly half our bonds went bust with no recovery, but our median bond is returning one percent!"