I've an unlimited amount of unlabelled data and an oracle that is able to label portions of this data using only two labels "1" or "0". I want to ask a few number of questions to the oracle and after a while I want to start to predict values with a certain confidence.
Ideally I want to minimize the number of user interactions so I want to use an active learning strategy in which I use a function f for predict the next unlabelled row to submit to the user. Because the user is able only to label with 1 or 0, I've opted for a logistic regression but any other suggestions would be appreciated. For the logistic regression I'm using the WEKA implementation.
Problem: At each user interactions I would estimate the confidence of the learned model (or the probability p that the model don't make mistakes in predictions). Ideally I want to state something like "for the next predictions the model is sure to make a correct prediction with a probability p".
Question: Is it feasible to estimate this kind of confidence?
My first idea it's to use a Wald test for each estimated coefficient but I've read that it doesn't work for small sample size (that is my condition). And sometimes could happen that it's hard to calculate the estimate variance of the beta coefficients because the matrix regression isn't invertible (I've used the procedure here for estimate the variance). Any suggestions on different kind of tests for estimating the "confidence" of the learned model would be appreciated.