1
$\begingroup$

I am familiar with the concept of combining forecasts. In the simplest scenario, that could be just averaging the forecasts. For example, let's say I have three forecasts representing the likelihood of an event such as (35, 50, 40). The combined forecast would be (35+50+40)/3 = 41.67 (although I realize there are some more sophisticated methods). This seems straightforward enough. Although I understand there are some risks associated with this, I was under the impression that combining forecasts is a common approach as in this question's answers.

However, let's say I have a situation that has multiple forecasts for multiple different groups (alternative options). As an intuitive example, a list of orders to be picked up from different locations.

order  prob  location
1      0.4   A
2      0.5   A
3      0.35  A
4      0.95  A
5      0.6   B
6      0.5   B
7      0.75  B 
8      0.15  B

Each order has a probability of being accepted (i.e. forecast). I want to say the probability of which location an order will be accepted. My initial instinct was to average the probabilities but I'm hesitant.

Note - the number of probabilities could exceed 100 per group and there could be >20 groups.

$\endgroup$
0
$\begingroup$

The combined forecast would be (35+50+40)/3 = 41.67

This is known as Simpson's paradox, a very dangerous approach to share values. The items you work with are probability estimations derived on samples of different sizes, and by averaging the share you will get a confusing result if the actual numbers of right / wrong answers are really different across groups.

It is better to make counts of right answers and divide them by total counts.

$\endgroup$
  • $\begingroup$ The intent is to use the results as a forecast, so the 'right' answers are not known. $\endgroup$ – cdeterman Oct 5 '17 at 15:17
  • $\begingroup$ OK, so I will clarify now what I meant. Averaging probabilites is a popular practice, as you would do when creating a forest of decision trees with binary output. I wanted instead to suggest using exact counts if you accumulated them. If those counts come from samples of different size, then instead of averaging probabilites (shares), you can divide "success" count (as you would do with a number of successes accumulated in terminal tree node) by total count for let's say location A. $\endgroup$ – Alexey Burnakov Oct 5 '17 at 15:43
  • $\begingroup$ So you are simply suggesting a weighted average by group size? $\endgroup$ – cdeterman Oct 5 '17 at 16:41
  • $\begingroup$ Yes, the weighted average according to a train sample size from which you derived the probabilities. $\endgroup$ – Alexey Burnakov Oct 5 '17 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.