# Measuring accuracy of estimates from lognormal distributions

Our org need to make estimates of movie box office results relative to our estimates pre-release.

We know that, generally, box office results are lognormally distributed. we can determine a good fit on a lognormal estimator and a large box office portfolio of actual results which matches pretty well.

My question has to do with discriminating the cause of errors in estimation of both individual estimates and portfolio total estimates.

E.g. if based on factors like budget cast director and genre and size of release we make an estimate of box office to be obtained and the amount of marketing spend to be made to support that estimate.
So if we estimate that a film will do 50MM in box office, and we spend marketing dollars accordingly, but the film only does 22MM, does that error look like an "outlier" (signalling that we were over-optimstic in our our estimation) or not? (or put another way, is there some p value we can measure against which says if our estimate is unbiased, then the actual result should be with x% of the estimate? (Assuming for example, that have set a target Mean Bias Error of, say, 20-30%) Or is there no reasonable way to make a judgement as to whether a single trial like this indicates anything about the overall bias of our "estimation engine" (e.g. a bunch of people sitting around talking)with respect to the prediction on a single trial?

Likewise, on a portfolio of say, 10 movies, how do we figure out if the delta between the portfolio estimate total box office and the portfolio actual total box office demonstrates that we are biased high in our estimates or not? On the portfolio case we measure simply the ratio of times we exceeded estimate and the times that we are short on the estimate as a measure of our bias, and feel OK if we were high roughly half the time and low roughly half the time, but I'm sure there is a better measure of our bias. However given we have only 10 history I wonder if that is enough to that the portfolio distribution should be symetric given the asymetry of the sampling distribution and the relatively low n. So would we expect that the say, 95% confidence interval should be smaller on the low side, and higher on the high side due to the asymetry of the log normal distribution?

Many thanks!

• You are missing an important piece of information here to determine how "far off" your estimate is: the standard deviation. When you say you estimate that the film will make 50mm, I presume you mean that, on average, a film with those given features will make 50mm. However, without knowing the standard deviation of that prediction, it is impossible to tell whether the observed \$22mm means your mean estimate (50mm) is likely wrong, or if it's just within what's expected given its variance. – user3208442 Apr 15 '18 at 20:09
• "The standard deviation of what?" I think is the relevant question. I think instead we want to use a target of some kind, and the target is the average error threshhold that is actually somehow useful to us for decision making. (if the standard deviation we are using is so big as to make just about any prediction look "unbiased" than we need to get better at prediciton or just give up the exercise entirely, as it provides no informaiton useful for decision making. Instead we talk about a target, ie. we want our mean bias error not to exceed some percentage, which is described above.. – GGizmos Apr 16 '18 at 18:15

One thing you can do is to compare the ability of several models (probably linear with lognormal link to outcome) to predict the actual outcome.

1. A model containing only the (logarithm of) expert assesment as predictor
2. A model containing only some objective values (e.g. budget, genre, ...) as predictor
3. A model containing all of the predictors

The difference in residual error is a measure of the amount of information the predictors add. Also in case of model 1) the intercept is a measure of bias.

Since your sample size is low, it might make sense to take a full Bayesian treatment which will work correctly even with small sample size (e.g. you will get a posterior distribution of the variables of interest and if your data do not contain a lot of information this will be reflected as a wide posterior distribution).

Further you may want to use a hierarchical model to pool some of the coefficients together and avoid overfitting. I made an example for a slightly similar case in another answer.

If you are using R, rstanarm might be a good choice covering all of the above.

Also, as pointed out in the comments, having some kind of deviation of your prediction might be useful and it is also IMHO at least a good excercise for experts to try to guess their uncertainty (most likely they will underestimate it).

EDIT: To answer the question fully: the only thing you can learn about a single prediction is its actual error - and you already know it, there is no statistics for single instances. Fitting the models I described will tell you something about the behavior portfolio-wide. If you had a lot of data, you might be able to investigate interactions, e.g. "are experts better at judging high budget films?", most modelling packages let you predict with interactions easily, but you need a lot of data to be able to estimate them reliably.

I could go on for a while, so please ask for details/clarifications in the comments if this line of reasoning is compelling to you.