Following up on "How to evaluate quality of probability estimator for Bernoulli experiments?", I want to visualize the quality of an estimator for probability forecasting using a Reliability Diagram. I do not however want to put the observations into "bins", which seems to me an unprecise estimation, but rather estimate the reliability using a smoothing process similar to Kernel Smoothing.
So, given $n$ observations $o_i \in \{0,1\}$ and $n$ predictions $\hat{p}_i \in [0,1]$, I want to estimate $f(x)=\frac{\sum_{i}o_i \cdot 1_{\{x\}}(\hat{p}_{i}(x))}{\sum_{i} 1_{\{x\}}(\hat{p}_{i}(x))}, x \in [0,1]$, that is the number of successes divided by the number of predictions for a probability value $x$. For a perfect predictor, $f(x)=x$.
What I am doing right now is trying to estimate $f$ by calculating $\hat{f}(x)=\frac{\sum_{i} K(\frac{x-x_i}{h}) \cdot o_i}{\sum_{i} K(\frac{x-x_i}{h})}$, where for $K$ I currently use the Epanechnikov Kernel. The remaining question is: how do I properly set the bandwidth parameter $h$? I am aware that there are methods to select an optimal bandwidth for Kernel Density Estimation, but since this is not KDE, I am not sure on how to proceed here.
