Distribution for simulating machine learning algorithm errors I want to simulate machine learning algorithm, that solves binary classification problem. This algorithm should output not just $\{0, 1\}$ labels, but probability of class $1$ (that of course is in $[0; 1]$ range). The question is what distribution should I choose for simulating errors produced by this algorithm. As I understand it should be some one-sided bounded distribution, but I don't know which one.
 A: If I understand correctly, you have some $p \in (0,1)$ and you would like to add some error $\epsilon$ to it such that $p+\epsilon \in (0,1)$.  The problem being, how to chose $\epsilon$ such that we don't escape the bounds $(0,1)$.
One solution may be to sample $\epsilon$ from any distribution with support over $\mathbb{R}$ (e.g Normal distribution), and add this error to the log-odds for $p$.  You then calculate your new $p^*$ to have the corresponding log odds.
To clarify, since $p \in (0,1)$,
$$
\ln\frac{p}{1-p}\in(-\infty,\infty)
$$
So we can sample our errors such that $\epsilon\in(-\infty,\infty)$ without changing the original range upon addition.  This gives us,
$$
\ln\frac{p^*}{1-p^*}=\ln\frac{p}{1-p}+\epsilon
$$
It can be shown that if
$$
\ln\frac{q}{1-q} = k 
$$
then,
$$
q = \frac{e^k}{1+e^k}
$$
in this case $q=p^*$ and $k=\ln\frac{p}{1-p}+\epsilon$, and so we have
$$
p^* = \frac{e^{\ln\frac{p}{1-p}+\epsilon}}{1+e^{\ln\frac{p}{1-p}+\epsilon}}
$$
$$
 = \frac{\frac{p}{1-p}e^\epsilon}{1+\frac{p}{1-p}e^\epsilon}
$$
Which will stay within $(0,1)$
Edit:
The last line can be further simplified to 
$$
 = \frac{e^\epsilon}{\frac{1-p}{p}+e^\epsilon}
$$
