Prediction Intervals on Judgment Forecasts - Possible? Are there any R packages available or general methodologies for calculating prediction intervals on multi-step judgmental forecasts?
I've looked at Hyndman's text and the R forecast package - which will provide intervals for each period in time, but also a point estimate - which may differ from the judgment forecast, therefore the intervals aren't necessarily applicable.
I read the portion of his book on judgment forecasting, which speaks to the pervasiveness of this technique, as well as the fact that its, of course, entirely sound (though maybe not the best approach) to create a judgment forecast based off of a combination of historical data, domain knowledge, knowledge of changes in the market, etc.
He also discusses generating forecasts via a model (naive, mean, etc.), then tweaking those forecasts further with judgment -- but he doesn't go on to discuss how the PI itself would also evolve.
Just curious if anyone has an experience or advice on this front - generating prediction intervals on judgment forecasts, any advice/suggestions are much appreciated.
Text for reference if interested: https://otexts.com/fpp2/index.html
 A: There are a number of methodologies for eliciting expert knowledge. E.g. one example is SHELF (that one e.g. has the associated SHELF R package), but there are also others like the Cooke performance-based method or Delphi. These try to obtain a full distribution, to varying degrees try to account for the usual human biases (e.g. too narrow uncertainty, anchoring etc.) and vary in complexity of implementation. I would guess any of those could be applied to specific forecasting problems.
If you directly asked for the forecasts of interest, you might be done after an elicitation. Otherwise, you may want to combine distributions for different parameters (based on elicitation, previous data/modeling and/or known information) through simulation (presumably would have to be bespoke for your problem) or Bayesian MCMC sampling (e.g. general MCMC sampler like Stan and the rstan package may serve you well here).
A: I see various possibilities.

*

*You could ask your judgemental forecasters to not only provide their "best guess", but also provide intervals that they are (say) 90% certain the actual will fall in. This is a frequently studied setup. Under laboratory conditions (so: no "real" forecasters, just undergrad students or Amazon Mechanical Turks), it invariably yields intervals that are too narrow. You could in principle post-process judgmental interval forecasts, but once your forecasters learn about this, they will game the post-processing.

*If you have system forecasts that do provide prediction intervals, you could shift the entire PI up or down by the difference between the judgmental forecast and the system expectation forecast. Essentially, if your system gives a 90% PI of $[1,10]$ with an expectation forecast of $3$, and your judgmental forecaster gives a forecast of $5$, then you just shift the system-generated PI up by $5-3=2$ to obtain a PI of $[3,12]$.

*If you have a sufficiently long history of judgmental forecasts and corresponding actuals, you can compare the two and get a distribution of the errors. You can then either directly take quantiles of the historical errors, or fit any distribution to the errors and take quantiles from that. Of course, this presupposes that your error distribution remains stationary.

