Measuring accuracy of estimates from lognormal distributions Our org need to make estimates of movie box office results relative to our estimates pre-release.
We know that, generally, box office results are lognormally distributed.
we can determine a good fit on a lognormal estimator and a large   box office portfolio of actual results which matches pretty well.
My question has to do with discriminating the cause of errors in estimation of both individual estimates and portfolio total estimates.
E.g. if based on factors like budget cast director and genre and size of release we make an estimate of box office to be obtained and the amount of marketing spend to be made to support that estimate.
So if we estimate that a film will do 50MM in box office, and we spend marketing dollars accordingly, but the film only does 22MM, does that error look like an "outlier" (signalling that we were over-optimstic in our our estimation) or not?    (or put another way, is there some p value we can measure against which says if our estimate is unbiased, then the actual result should be with x% of the estimate?  (Assuming for example, that have set a target Mean Bias Error of, say, 20-30%)  Or is there no reasonable way to make a judgement as to whether a single trial like this indicates anything about the overall bias of our "estimation engine" (e.g. a bunch of people sitting around talking)with respect to the prediction on a single trial?
Likewise, on a portfolio of say, 10 movies, how do we figure out if the delta between the portfolio estimate total box office and the portfolio actual total box office demonstrates that we are biased high in our estimates or not?   On the portfolio case we measure simply the ratio of times we exceeded estimate and the times that we are short on the estimate as a measure of our bias, and feel OK if we were high roughly half the time and low roughly half the time, but I'm sure there is a better measure of our bias.  However given we have only 10 history I wonder if that is enough to that the portfolio distribution should be symetric given the asymetry of the sampling distribution and the relatively low n.  So would we expect that the say, 95% confidence interval should be smaller on the low side, and higher on the high side due to the asymetry of the log normal distribution?
Many thanks!
 A: One thing you can do is to compare the ability of several models (probably linear with lognormal link to outcome) to predict the actual outcome.


*

*A model containing only the (logarithm of) expert assesment as predictor

*A model containing only some objective values (e.g. budget, genre, ...) as predictor

*A model containing all of the predictors


The difference in residual error is a measure of the amount of information the predictors add. Also in case of model 1) the intercept is a measure of bias.
Since your sample size is low, it might make sense to take a full Bayesian treatment which will work correctly even with small sample size (e.g. you will get a posterior distribution of the variables of interest and if your data do not contain a lot of information this will be reflected as a wide posterior distribution). 
Further you may want to use a hierarchical model to pool some of the coefficients together and avoid overfitting. I made an example for a slightly similar case in another answer.
If you are using R, rstanarm might be a good choice covering all of the above.
Also, as pointed out in the comments, having some kind of deviation of your prediction might be useful and it is also IMHO at least a good excercise for experts to try to guess their uncertainty (most likely they will underestimate it).
EDIT: To answer the question fully: the only thing you can learn about a single prediction is its actual error - and you already know it, there is no statistics for single instances. Fitting the models I described will tell you something about the behavior portfolio-wide. If you had a lot of data, you might be able to investigate interactions, e.g. "are experts better at judging high budget films?", most modelling packages let you predict with interactions easily, but you need a lot of data to be able to estimate them reliably. 
I could go on for a while, so please ask for details/clarifications in the comments if this line of reasoning is compelling to you.
