1
$\begingroup$

I've an unlimited amount of unlabelled data and an oracle that is able to label portions of this data using only two labels "1" or "0". I want to ask a few number of questions to the oracle and after a while I want to start to predict values with a certain confidence.

Ideally I want to minimize the number of user interactions so I want to use an active learning strategy in which I use a function f for predict the next unlabelled row to submit to the user. Because the user is able only to label with 1 or 0, I've opted for a logistic regression but any other suggestions would be appreciated. For the logistic regression I'm using the WEKA implementation.

Problem: At each user interactions I would estimate the confidence of the learned model (or the probability p that the model don't make mistakes in predictions). Ideally I want to state something like "for the next predictions the model is sure to make a correct prediction with a probability p".

Question: Is it feasible to estimate this kind of confidence?

My first idea it's to use a Wald test for each estimated coefficient but I've read that it doesn't work for small sample size (that is my condition). And sometimes could happen that it's hard to calculate the estimate variance of the beta coefficients because the matrix regression isn't invertible (I've used the procedure here for estimate the variance). Any suggestions on different kind of tests for estimating the "confidence" of the learned model would be appreciated.

$\endgroup$
  • $\begingroup$ I think you're on the right track. Some type of sequential probability ratio test (SPRT). $\endgroup$ – HEITZ Oct 3 '16 at 22:09
  • $\begingroup$ I want to do something like Likelihood-ratio test, but in my settings I haven't two models one for the null hypothesis and another one for the alternative hypothesis. I can't compute the log likelihood. $\endgroup$ – Enzo Oct 3 '16 at 23:51
  • $\begingroup$ On Wald - Scott Menard's free online logistic regression book explains more about why one should avoid relying on those statistics. $\endgroup$ – rolando2 Jan 18 '17 at 20:19
1
$\begingroup$

In classification setting, there are two popular query strategies: uncertainty sampling and query by committee (see paper for an extensive review). In uncertainty sampling, an active learner queries the label about which it is least certain. For example, we can choose to query a point that has maximum entropy (computed from class probabilities). On the other hand committee strategies consists of a set of classifiers and the most informative query is considered to be a point about which the committee disagrees the most. Such a point can be computed by maximizing vote entropy or average KL divergence. I recommend to review the paper above for additional query strategies.

If you are looking for implementations, consider the following library. It is compatible with scikit-learn and can be used with any classifier. It uses random subsampling as a baseline for measuring the benefit of active learning.

$\endgroup$
1
$\begingroup$

Question: Is it feasible to estimate this kind of confidence?

In general, the idea is reasonable in my opinion. However, especially in an active learning setting with its naturally incurred sampling bias to the model, you have to consider what such a confidence measure means - a notion of confidence of your model, based on the data it has seen so far.

Of course, the models uncertainty is one of the factors considered in active learning literature. With probabilistic models it can directly be taken from the model, while with SVMs for example, the distance to the decision hyperplane(s) serves as a proxy.

But what does the confidence/uncertainty of the model tell us? It gives us a measure of how confident we can be in the models predictions. Consider the example below: with the two classes blue & red, the circles indicating labeled instances and the small Xs indicating unlabeled data - how confident are we in the models prediction? AL scatterplot

If we estimate how well our model explains the labeled data we have sampled so far (here with uncertainty sampling and random instances as initialization) we will probably be happy with the result.

However, without any notion of how well our sampling actually covers the available data, we might be deceived by the confidence in the model.

Of course, in the sketched example it is easy to see - with data of higher dimensionality and less nicely separated clusters, this is hardly possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.