Following up on "How to evaluate quality of probability estimator for Bernoulli experiments?", I want to visualize the quality of an estimator for probability forecasting using a Reliability Diagram. I do not however want to put the observations into "bins", which seems to me an unprecise estimation, but rather estimate the reliability using a smoothing process similar to Kernel Smoothing.

So, given $n$ observations $o_i \in \{0,1\}$ and $n$ predictions $\hat{p}_i \in [0,1]$, I want to estimate $f(x)=\frac{\sum_{i}o_i \cdot 1_{\{x\}}(\hat{p}_{i}(x))}{\sum_{i} 1_{\{x\}}(\hat{p}_{i}(x))}, x \in [0,1]$, that is the number of successes divided by the number of predictions for a probability value $x$. For a perfect predictor, $f(x)=x$.

What I am doing right now is trying to estimate $f$ by calculating $\hat{f}(x)=\frac{\sum_{i} K(\frac{x-x_i}{h}) \cdot o_i}{\sum_{i} K(\frac{x-x_i}{h})}$, where for $K$ I currently use the Epanechnikov Kernel. The remaining question is: how do I properly set the bandwidth parameter $h$? I am aware that there are methods to select an optimal bandwidth for Kernel Density Estimation, but since this is not KDE, I am not sure on how to proceed here.

  • 2
    $\begingroup$ You have to select the criterion you want to minimise/maximise and the find the optimal bandwidth. An alternative is using automatic nonparametric estimation, such as the one proposed here. The R packages LogConcDEAD and logcondens might be of interest if you find this approach appealing. $\endgroup$ – user10525 May 27 '12 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.