4
$\begingroup$

I have 3 independent sources for tomorrow's weather forecast:

  1. 100% probability for snow, this source is 80% accurate
  2. 50% probability for snow, this source is 60% accurate
  3. 0% probability for snow, this source is 40% accurate

The accuracy for each source is: $$\frac{\text{Number of correct forecasts}}{\text{Total number of forecasts}}$$ What is the best estimate for the probability for snow tomorrow?

Essentially it is an extension of a previous question, but where each source has a different accuracy or reliability.

Also, the selected answer suggested using the geometric mean, ignoring the fact that one of the probabilities is 0, collapsing the entire answer to 0 which intuitively makes no sense: the forecast with the lowest accuracy should not supersede a more accurate forecast just because of numerical considerations.

My intuition was to solve it weighing probabilities with accuracies: $$\frac{1.0 \times 0.8 + 0.5 \times 0.6 + 0 \times 0.4} {0.8 + 0.6 + 0.4}$$ but the interviewer insisted on solving using Bayes theorem.

In addition, if one of my sources has accuracy 100%, then it makes no sense to calculate a weighed mean with the other sources.

I could weigh as in AdaBoost:
$$\alpha_m = \frac{1}{2}\ln\left( \frac{1 - \epsilon_m}{\epsilon_m}\right)$$ so e.g. for the source with accuracy 80%, the weight would be
$$\alpha_m = \frac{1}{2}\ln\left( \frac{0.8}{0.2}\right) = \frac{1}{2}\ln(4)$$ etc. Is this an acceptable solution for this question?
In any case I'd be very happy to see how it can be solved using Bayes.

I have seen a few other questions similar to it, but none exactly the same.

$\endgroup$
  • 1
    $\begingroup$ The problem is interesting, but the wording is a bit confusing in my opinion. What does it mean that source Y is $x%$ accurate? It cannot mean that the probability of snowing $P$ is a discrete random variable, which is equal to 0 with probability 0.4, to 0.5 with probability 0.6 and to 1 with probability 0.8, because the sum of the probability masses would be > 1. So another model must be considered, but the wording makes it harder to understand which one... $\endgroup$ – DeltaIV Apr 16 '17 at 14:51
  • $\begingroup$ @DeltalV These are independent sources, so for each one, accuracy=(number correct forecasts)/(total number of forecasts). I'm editing the question to make it clearer. $\endgroup$ – carmi Apr 16 '17 at 14:59
  • $\begingroup$ But accuracy is about classification and here we have regression, so some additional loss function must be specified, but then it's hard to call it accuracy? Or maybe we have to consider only 3 classes, that is 0%, 50% and 100% probability of snow? $\endgroup$ – Łukasz Grad Apr 16 '17 at 15:13
  • 1
    $\begingroup$ The event may be binary, but since the forecasts are probabilistic, typically "accuracy" would be more complicated than "# correct/# forecasts" (see here). Also, the suggested answer was geometric mean of probabilities or odds-ratios? (i.e. arithmetic mean of logits) $\endgroup$ – GeoMatt22 Apr 16 '17 at 15:19
  • 2
    $\begingroup$ This question makes no (Bayesian) sense to me unless 'accuracy' is better defined. $\endgroup$ – Memming Apr 16 '17 at 20:19
3
$\begingroup$

With probabilistic forecasts, accuracy cannot be simply defined as (number correct forecasts)/(total number of forecasts), or better, you must define what you mean by a "correct forecast". For example you can set a threshold: if your forecast gives P("snow")>0.5, you predict "snow", otherwise "no snow". With this threshold, then if it snows tomorrow, source 1's forecast is correct, and the other two are wrong. At this point, you can use the same procedure as in question

https://stats.stackexchange.com/a/34141/58675

to compute a probability of snow tomorrow, conditional to the fact that source 1 predicts "snow", and source 2 and 3 don't ($P(y=1|\hat{y}_1=1,\hat{y}_2=0, \hat{y}_3=0)$. However, with respect to the original question, we have already introduced an arbitrary threshold. Not only that: as you can see in the linked answer, to get $P(y=1|\hat{y}_1=1,\hat{y}_2=0, \hat{y}_3=0)$ we need also the prior probabilities $P(y=1), P(y=0)$. Assignign these quantities would be arbitrary too. Thus, I don't believe your question has a unique answer, if we have to rely on Bayes' theorem.

$\endgroup$
  • 2
    $\begingroup$ I wish I had thought about this during the interview! This just goes to show how deep understanding of statistics is lacking in CS graduates, even with a PhD... $\endgroup$ – carmi Apr 18 '17 at 7:15
1
$\begingroup$

If you were using Bayesian Model Averaging, you would average your predictions $p(y_{t+1}|m_k,y_{1:t})$ with weights that represented some posterior model probabilities $p(m_k|y_{1:t})$ like this $$ p(y_{t+1}|y_{1:t}) = \sum_k p(y_{t+1}|m_k,y_{1:t})p(m_k|y_{1:t}). $$

One way I could understand your question is to find under what circumstances can we understand your "accuracy" as a posterior model probability. If you were updating your model's posterior probabilities at every time step using this recursive formula $$ p(m_k|y_{1:t}) = \frac{p(y_{t}|m_k)p(m_k|y_{1:t-1})}{\sum_{k'} p(y_{t}|m_{k'})p(m_{k'}|y_{1:t-1}) }, $$ then the posterior would increase after correct predictions, and decrease with bad predictions. If you started off with uniform priors over all of your models, and if you only had two classes for the categorical observations, and if you predict $Y_t = 1$ when $P(Y_t =1|m_k) > .50$, and if your data was iid, then your "accuracy" could represent something close to the posterior model probabilities. It would probably get pretty close after a few steps.

$\endgroup$
  • $\begingroup$ I think your answer is wrong. The OP's accuracies can't represent anything close to posterior model probabilities (except if for "close" you mean they're both real numbers in [0,1]!). Posterior model probabilities sum to 1 (over the space of all possible models): OP's accuracies sum to 1.8(!). $\endgroup$ – DeltaIV Apr 17 '17 at 7:49
  • $\begingroup$ @DeltaIV then we can just normalize like OP did In his/her answer $\endgroup$ – Taylor Apr 17 '17 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.