I've been seeing the term "benchmark" being thrown around when referring to ARIMA models (e.g. here: https://robjhyndman.com/hyndsight/benchmarks/). I've looked up a definition and cannot find one.
What does "benchmark" mean and why would I want to have it?
This is not specific to arima, but to all forecasting methods - and indeed to any kind of prediction exercise.
The idea is that it is always easy to cook up an enormously complex and very impressive forecasting/prediction method. The hard part is showing that this shiny contraption actually improves on the forecasts/predictions of established methods. Because if it doesn't, then the wonderful new method is not worth a lot (except for the aspect that we learn from our mistakes). In the post you link to, Rob Hyndman as the editor-in-chief of the International Journal of Forecasting notes that he will reject any submitted manuscript proposing a new forecasting method that does not do such a comparison.
These established methods are benchmarks.
For instance, it is not unusual for extremely simple methods like the overall mean or median to outperform ARIMA, so these very simple methods should always be used as benchmarks. If your new method cannot even improve on the overall mean, it's probably not all that good.
Similarly, Rob and George Athanasoupoulos called for submissions to a tourism forecasting competition a while back, and since they are both quite capable of fitting ARIMA models, they required that submitted forecasting methods outperform the MASEs of such ARIMA models. In this case, the benchmark is the automatically-fitted ARIMA model.
Benchmark is a very simple model. I don't know of any standard definition, and I've seen the term applied multiple times with very different meanings. For example, in this competition, for each timeseries, they used the last two points to create a trendline. Here they use the median per series, and here they use S&P500 as benchmark (which almost no one beats).
