Active learning using logistic regression I've an unlimited amount of unlabelled data and an oracle that is able to label portions of this data using only two labels "1" or "0". I want to ask a few number of questions to the oracle and after a while I want to start to predict values with a certain confidence.
Ideally I want to minimize the number of user interactions so I want to use an active learning strategy in which I use a function f for predict the next unlabelled row to submit to the user.
Because the user is able only to label with 1 or 0, I've opted for a logistic regression but any other suggestions would be appreciated. For the logistic regression I'm using the WEKA implementation.
Problem: At each user interactions I would estimate the confidence of the learned model (or the probability p that the model don't make mistakes in predictions). Ideally I want to state something like "for the next predictions the model is sure to make a correct prediction with a probability p".
Question: Is it feasible to estimate this kind of confidence?
My first idea it's to use a Wald test for each estimated coefficient but I've read that it doesn't work for small sample size (that is my condition). And sometimes could happen that it's hard to calculate the estimate variance of the beta coefficients because the matrix regression isn't invertible (I've used the procedure here for estimate the variance). Any suggestions on different kind of tests for estimating the "confidence" of the learned model would be appreciated.
 A: In classification setting, there are two popular query strategies: uncertainty sampling and query by committee (see paper for an extensive review). In uncertainty sampling, an active learner queries the label about which it is least certain. For example, we can choose to query a point that has maximum entropy (computed from class probabilities). On the other hand committee strategies consists of a set of classifiers and the most informative query is considered to be a point about which the committee disagrees the most. Such a point can be computed by maximizing vote entropy or average KL divergence. I recommend to review the paper above for additional query strategies. 
If you are looking for implementations, consider the following library. It is compatible with scikit-learn and can be used with any classifier. It uses random subsampling as a baseline for measuring the benefit of active learning.
A: 
Question: Is it feasible to estimate this kind of confidence?

In general, the idea is reasonable in my opinion. However, especially in an active learning setting with its naturally incurred sampling bias to the model, you have to consider what such a confidence measure means - a notion of confidence of your model, based on the data it has seen so far. 
Of course, the models uncertainty is one of the factors considered in active learning literature. With probabilistic models it can directly be taken from the model, while with SVMs for example, the distance to the decision hyperplane(s) serves as a proxy. 
But what does the confidence/uncertainty of the model tell us? It gives us a measure of how confident we can be in the models predictions. Consider the example below: with the two classes blue & red, the circles indicating labeled instances and the small Xs indicating unlabeled data - how confident are we in the models prediction?
 
If we estimate how well our model explains the labeled data we have sampled so far (here with uncertainty sampling and random instances as initialization) we will probably be happy with the result. 
However, without any notion of how well our sampling actually covers the available data, we might be deceived by the confidence in the model.
Of course, in the sketched example it is easy to see - with data of higher dimensionality and less nicely separated clusters, this is hardly possible.
