Performance evaluation of auto.arima in R and UCM on one dataset I started evaluating and comparing some methods in forecasting. I used Price of dozen eggs in US, 1900–1993, in constant dollars in the R software FMA package. I held out the last 10 years for assessment of forecast. Below are the results:
I used auto arima method in the R software. Obviously the results are way off. Am I doing something incorrect ? Below is the forecast. It does not recognize the declining trend. 

I also used an unobserved components model (UCM) and obtained a good forecast,  as below.


*

*Without outliers/level shifts there are very large standard errors and therefore wide confidence bands. 

*After some iterative work, below is the output with outliers/level shifts (I know I'm overfitting here) but it did a pretty good job in forecasting; there are also narrow confidence bands. 
In looking at just this example the UCM seems to predict the hold-out sample more accurately than auto.arima.
Why is auto.arima not providing a reasonable forecast?
Are state space models/UCMs better for forecasting long range?
Are there any benefits of using one method over other?
 A: A recent change to the way regression coefficients are initialized in the estimation of ARIMA models means that a different model is now selected by auto.arima (using R3.0.2):
> auto.arima(window(eggs,end=1983))

ARIMA(0,1,0) with drift         

Coefficients:
        drift
      -2.2665
s.e.   3.1133

sigma^2 estimated as 804.5:  log likelihood=-390.15
AIC=784.31   AICc=784.46   BIC=789.14

See https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=15396 for discussion of the change in the initialization. This is relevant here as the drift is estimated as a linear regression. The change is in stats::arima, not in auto.arima.
Note that the drift is not significant in any case. So you can hardly claim that the non-trended model is unreasonable.
Of course, if you know something about the data, then you should use the knowledge you have. But if you want something completely automatic, you can't really complain if it gives you something that doesn't fit your preconceived ideas of what is reasonable.
A state space unobserved component model will be very similar to an ARIMA model for these data. In fact, there will be an equivalent ARIMA model for any sensible UCM fitted to the data. So it makes no sense to say one forecasts better than the other. One might be easier to use than the other, or more interpretable than the other. 
Yes, there are benefits. ARIMA is much better at easily handling complicated short-term dynamics. UCM is much better at decomposing the series into interpretable components.
A: I employed AUTOBOX ( a piece of software that I have helped develop ) . The automatic model identification scheme detect a first difference model with an ar(3) component. . The test for constancy of parameters revealed a possible breakpoint at or around period 53 (year=1953 note that the UCM model declared a new trend at 1954) which suggests a possible regime change between 1-52 AND 53-83 . This is easily confirmed visually by examining a plot of the data. It is an exponential smoothing model ( a very particuLlar case of an ARIMA MODEL ) with a constant and an adjustment for the 75th data point. The residual plot is suggestive of sufficiency (at least with 31 values)  with an ACF of  . The next plot is the actuals/fit and forecast . 
The ARIMA model identified has a significant negative constant and thus provides downward guidance. The difference between AUTOBOX other approaches is that AUTOBOX tested for transient parameters and concluded that the older data (obs 1-52) was inconsistent with observations 53-83. This is the "elephant in the room" that nobody dares to mention.  The assumption that all of the data comes from the same model with constant parameters needs to be verified NOT ignored. Just because we know that 83 values exist DOESN'T mean that we should use all of the data. Modelling the entire series does not necessarily model individual subsets. 
I must comment that evaluating a forecast from 1 origin is insufficient research . One must look at many origins and for possibly different lead times. 
A: I don't get the same result as the OP (version 5.0 of the forecast package). If you run the following, the result is indeed a linear downward trend.
install.packages("fma")
library(fma)
install.packages("forecast")
library(forecast)

#window as per the OP
eggs2<-window(eggs, start=c(1900), end=c(1983))   
plot(eggs2)

#does produce a linear trend downward!
model2<-auto.arima(eggs2)
model2
plot(forecast(model2,10))

Maybe there was a change where allowdrift was not TRUE by default in the package used?

Further, if you run auto.arima on the full data set (without withholding the last 10 years) which is I believe what Dr. Hyndman was doing, the down ward trend is picked up.
model<-auto.arima(eggs)
model

Series: eggs 
ARIMA(0,1,1) with drift         

Coefficients:
          ma1    drift
      -0.1630  -2.3774
s.e.   0.1145   2.3229

sigma^2 estimated as 713:  log likelihood=-432.26
AIC=870.51   AICc=870.78   BIC=878.11

