Which model is better: One that overfits or one that underfits? From practical point of view, is it better to have a model that is overfitting the data or a model that is underfitting it?
 A: Compare the two models on the plot below. First (red curve) heavily overfitts the data, while the second one (blue line) underfitts it. Can you tell which one is better? I'd say that both are bad.

If your model overfitts, then it perfectly fits your data, but can possibly be poor for making out-of-sample predictions. If it underfitts, then if poorly fits your in-sample data, but it can work better or worse on external data. Both cases are bad. From practical point of view you choose the model that best fits your purpose, or saying it in two words, it depends. From practical point of view you wouldn't be considering model that is poor.
A: It's hard to say in general whether overfitting or underfitting is less desireable — this probably depends on your application.
In practice though, it's probably better to start with a model that has enough capacity to overfit your training data since there are lots of techniques for dealing with overfitting (e.g., regularization, dropout, dimensionality reduction, etc.)
A: In ARIMA model over-fitting is often performed due to model mis-specifcation incorporating an unwarranted differencing operator and a self-cancelling ma operator leads to poor forecasts. Incorporating self-cancelling ARMA structure also has the same effect. In terms of incorporating statistically non-significant (unnecessary i.e. not different from 0.0) lag structures for causal variables generally has little or no effect. I suggest that over-modelling ARIMA models can be a bad idea while under-modelling casual models is a bad idea.
