4

I can recommend you Yuxin Chen and Andreas Krause's NIPS'13 paper about Near-optimal Batch Mode Active Learning and Adaptive Submodular Optimization. Reformulating your hypothesis tests using the Adaptive Submodularity framework provides great theoretical run-time guarantees for greedy algorithms. In this paper, they showed that their hit-and-run sampler ...


2

To answer my own question, the optimal way to pick an initial sample according to information criteria such as entropy is a notorious problem called maximal entropy sampling. This turns out to be NP-hard, so I will probably select a small uniform sample of the data and then try to apply maximal entropy sampling afterwards. For approximations, this post ...


2

Something that reminds me of this is the framework of generative adversarial networks. These models have two parts: a generator, which is usually implemented as a neural network mapping uniformly distributed inputs to the data you generate, e.g. images, and a discriminator, which tries to classify between generated samples and true data. The generator is ...


1

The principals of model selection are the same when using active learning as with standard classification. You still want to make model selection decisions using separate data (e.g. validation set or cross-validation) from the training data to avoid overfitting of the hyperparameters. In the case where labeled data is very rare, leave-one-out cross-...


1

Assuming you just automatically get the correct labels for the test set after making a prediction, this is a standard problem known as online learning. Just incorporate the new labels into your training set and update your model. For standard modern models learned via SGD, as long as you think your test set is approximately IID from the target distribution, ...


1

While I'm not aware of any tutorials, I can provide a bit of advice from my past experience. Research regarding active learning has tackled many different aspects of this domain, but not all of them might be relevant to you. Start with rather simple approaches, like alternating random sampling and uncertainty sampling in an appropriate interval. It is a ...


1

Question: Is it feasible to estimate this kind of confidence? In general, the idea is reasonable in my opinion. However, especially in an active learning setting with its naturally incurred sampling bias to the model, you have to consider what such a confidence measure means - a notion of confidence of your model, based on the data it has seen so far. Of ...


1

In classification setting, there are two popular query strategies: uncertainty sampling and query by committee (see paper for an extensive review). In uncertainty sampling, an active learner queries the label about which it is least certain. For example, we can choose to query a point that has maximum entropy (computed from class probabilities). On the other ...


1

martinthenext! I am not sure about the peculiarity of your problem, but if it's really crucial for you not to show -1 samples to a user, maybe you would better have a look at so-called "One-class classification", carefully described in a publicly available paper. SVM method is presented there, but as I understand, it can be generalized for any metric ...


Only top voted, non community-wiki answers of a minimum length are eligible