We have a data-point x and many classes. Let $P(c|x)$ the probability that $x$ is of class $c$. We note $c_1$ the most probable class for $x$ (i.e. $P_1=P(c_1|x)$ is the highest probability), $c_2$ the second most probable class for $x$ (i.e. $P_2=P(c_2|x)$ is the second highest probability ($P_1> P_2$)).
Obviously, when $P_1$is close to P2 (i.e. $P_1P_2$ is small or closer to 0) we are not sure whether $c_1$ or $c_2$is the true class of $x$, then it is useful to ask for the true label of $x$.
We define $u$ to be a threshold value for the difference between the highest and the second highest probability (a threshold for $P_1$-P2); thus, if $P_1-P_2 > u$ then we can (with some cost) ask for the true class of $x$ (let's note it $c_x$).
$c_1$(the predicted class for $x$) is usually equals to cx, but may sometimes not be equal. Given this problem, I want to learn a good value for the parameter $u$. To do that, currently, I just set u at an initial value (e.g. $u=0.2$) and then adjust this value according to whether or not $c_1$ equals $c_x$:
if $c_1 = c_x$then we get more confident and thus decrease the value of $u$ (e.g. $u=u-\epsilon$), otherwise (when $c_1$!= cx) we get less confident and thus increase the value of u (e.g. $=u+\epsilon$), where $\epsilon=0.01$ for example.
Is there any better why to "learn" a value for $u$ ? (Assuming that we can start with a hight initial value of u in order to get labelled data at the beginning, or assuming that I have a subset of labelled data that I can to use to learn the value of $u$).