This is not specific to arima, but to all forecasting methods - and indeed to any kind of prediction exercise.
The idea is that it is always easy to cook up an enormously complex and very impressive forecasting/prediction method. The hard part is showing that this shiny contraption actually improves on the forecasts/predictions of established methods. Because if it doesn't, then the wonderful new method is not worth a lot (except for the aspect that we learn from our mistakes).
In the post you link to, Rob Hyndman as the editor-in-chief of the International Journal of Forecasting notes that he will reject any submitted manuscript proposing a new forecasting method that does not do such a comparison.
These established methods are benchmarks.
For instance, it is not unusual for extremely simple methods like the overall mean or median to outperform ARIMA, so these very simple methods should always be used as benchmarks. If your new method cannot even improve on the overall mean, it's probably not all that good.
Similarly, Rob and George Athanasoupoulos called for submissions to a tourism forecasting competition a while back, and since they are both quite capable of fitting ARIMA models, they required that submitted forecasting methods outperform the MASEs of such ARIMA models. In this case, the benchmark is the automatically-fitted ARIMA model.