# Most probable value given predictions with confidence levels

I'm new to http://stats.stackexchange, so please let me know if I'm doing something wrong.

I have the following problem:

Suppose that there is a car traveling along a flat surface. There are no roads on this surface, so the travel can be thought of as random.

There are five people in a helicopter flying above this surface, all of whom are observing this car from the helicopter, through the same window (in the helicopter).

After some time, each person is asked to predict where the car will be in one second. Each person gives a prediction, along with how confident they are of that prediction i.e. person P says "I am C% confident that in one second from now, the car will be at location L).

Given that there are five such people, how can I determine the most probable location of the car one second from now?

I don't think this really matters, but being a stats n00b, I'd rather provide this information than not: the number of people in the helicopter is variable. Five is just an example for this case. It's not fixed.

The observers in the helicopter have all been trained for such a prediction task. However, each observer specializes in predicting certain types of behavior. For example, one observer might be trained in predicting all types of left turns, etc.

Assume that human error does not exist and that all observers tell the truth. Also assume that they all have the same objective scale of measuring their own confidences.

A note on confidences: Confidence does not refer to a confidence interval, which talks about the range of possible/probable values for a variable. A bad example (but one that illustrates the underlying point): if observer A says "I will bet \$100 on my prediction being correct" and observer B says "I will bet \$150 on my prediction being correct", all other factors (such as how much each observer values their money) kept constant, observer B is more confident than observer A. Assume that all observers have been properly trained to correctly use a particular objective scale by which to measure their confidence levels.

-

If everything is reasonable (and an easy way to check this on real data will be to see that the 5 predictions are fairly close), you can take a weighted average with weights proportional to the confidence levels.

If the predictions are very different, you'll have to get some idea why; maybe one of the 5 is an overconfident idiot, maybe the travel isn't as random as you think.

You can probably do better if you have a more precise understanding of what "I am C% confident that in one second from now, the car will be at location L" actually means, like what is the probability distribution (say, a Gaussian around L?), and where exactly the number C comes from.

-
Could you please explain the theoretical justification for a weighted average and for choosing weights proportional to the confidence levels? Please bear in mind that under the standard understanding of "confidence" in statistics, higher confidence must be associated with a larger region in which the car could be located, so that it's likely these people are not using "confidence" in the same sense--and probably not in any consistently quantitative sense in the first place. –  whuber Mar 7 '12 at 18:33
@whuber I don't mean confidence intervals, which is what I think would imply a larger region. I mean that the person is very confident about his prediction. I'm going to give a bad example, but the underlying point is valid: a guy who says "I'd bet my life on my prediction" is more confident than a guy who says "I'd bet \$100 on my prediction", assuming that they both value their life and \$100 the same as the other. –  inspectorG4dget Mar 7 '12 at 18:45
@AVB: to make this more interesting, let's say that the predictions of the five people can be quite different from one another. I need to pick the prediction which is most likely true. –  inspectorG4dget Mar 7 '12 at 18:46
You need to operationalize "confidence" then. This is usually done by (a) training the observers so they can be accurate and precise and (b) calibrating each observer so you know just how inaccurate and imprecise they are. Without this information, you're just making up a solution and hoping it might work. (If Mitt Romney is on that helicopter and bets you \$10,000,000 that he's right, are you going to put all your weight on his estimate? What does his wealth--or for that matter, his "confidence"--have to do with his observational abilities?) –  whuber Mar 7 '12 at 18:48
OK, Inspector, why don't you add this information to your question? Specifically, it now sounds like you have data telling us how good the predictors are. If you put that into evidence--or at least tell us its nature--we would have a better chance to provide objective, justified answers rather than just guessing at things that might work. (I think AVB is on the right track--so to speak--with the Kalman filter suggestion.) –  whuber Mar 7 '12 at 18:58