Skip to main content

Questions tagged [active-learning]

Active learning is a setting where an automated learning system can request labels from an external source, perhaps a human user or a real-world experiment. It is used to try to learn good models while minimizing the number of interventions required.

Filter by
Sorted by
Tagged with
1 vote
0 answers
9 views

Active learning for simulation parameters

Suppose we are training a neural network (or more generally, your favorite nonparametric model) $f: \mathbb{R}^k \to \mathbb{R}$ to solve a regression problem. For clarity, $k$ is of the order $\sim10$...
mikefallopian's user avatar
0 votes
0 answers
11 views

What is the relationship between maximum entropy sampling and bayesian d-optimal design?

In regards to optimal design, specifically sequential design or active learning for parameter inference (e.g. linear regression). As I understand it, Bayesian D-optimal design maximises the expected ...
Septimus Boshoff's user avatar
2 votes
1 answer
80 views

Best interpolating points for a Gaussian process regression

I have an unknown function $f(x)$, defined on a domain, that is modeling a perception function based on a human user response. I estimate it with a GP with mean $\mu$ and kernel $K$. I determine the ...
Andrea Gulli's user avatar
0 votes
0 answers
36 views

Best function value for Expected Improvement computation for Gaussian Process Regression

I want to implement Gaussian Process regression in the context of active learning, in which interpolation is performed with the best interpolating points, selected at each step iteratively. At every ...
Andrea Gulli's user avatar
1 vote
0 answers
23 views

Active learning for Computer Vision where I have to generate my own images

I'm interested in applying active learning to a computer vision bounding box prediction project, where I don't have a large corpus of unlabeled images available, and instead have to take all pictures ...
gkml88's user avatar
  • 11
3 votes
1 answer
55 views

Statistical test to determine if Active Learning has provided a significant improvement?

I am conducting various active learning experiments on two Biomedical Relation Extraction corpora: 2018 n2c2 challenge: 41000 test samples DDI Extraction corpus: 5700 test samples and using four ...
pabvvv's user avatar
  • 31
0 votes
0 answers
20 views

Can models improve the more you use them?

I came across this interesting question on Quora (someone else asked): How can I penalize a deployed classifier for every wrong prediction in the production run? I want to implement an online model ...
stochastic_caloric_descent's user avatar
0 votes
1 answer
45 views

Name and origin of a metric for evaluating ranking performance

I came across an evaluation metric to test whether a predicted rank is good, especially the top k items. But I don't know what it is called or where it is used, which makes my discussion of this ...
Iris Wang's user avatar
0 votes
1 answer
41 views

How to adaptively select stimuli in paired comparison experiment?

I have 500 items of which I want to know how much they are liked by a specific expert. To find how much they are liked, I wish to use paired comparisons since this yields more accurate results than ...
Max J.'s user avatar
  • 113
1 vote
0 answers
94 views

How to interpret the relationship between batch size and bootstrap count in a specific paper?

In the paper "Active Learning for Natural Language Parsing and Information Extraction", the author mentioned: In tests on this data, test examples were chosen independently for 10 trials ...
LCheng's user avatar
  • 189
1 vote
0 answers
18 views

Reasoning about increased performance from two disjoint datasets compared to their union

Say I have some pretrained model with an ok but not great accuracy. I can augment my existing training data with new datasets and retrain to increase performance. Say there are two datasets available, ...
Safwan Hossain's user avatar
2 votes
0 answers
28 views

Which item to administer to improve the standard error of the item difficulty estimate the most?

I have a student with a known ability level and a set of items with an estimated difficulty. The probability of a student getting an item correct follows an 1 parameter logistic model (also known as ...
rep_ho's user avatar
  • 7,649
2 votes
1 answer
144 views

Query-By-Committee with abstention

I’ve some difficulties understanding how abstention works in Active Learning. A teacher asked me to implement the active learning algorithm Query-by-Committee which helps a committee to ask the better ...
MarcAntony's user avatar
2 votes
0 answers
33 views

Active learning to counter concept drift

I'll be doing my thesis soon on model drift detection and possible remedies in a production environment. I'll probably be making an intuitive (hopefully!) theoretical framework with various types of ...
Zestar75's user avatar
1 vote
0 answers
21 views

Is variance of individual points the best strategy for chosing points in active learning?

In active learning with estimators like Gaussian Processes, we typically look at the test-test covariance matrix, and choose for the next round of observation, the points who have the largest ...
user3246971's user avatar
1 vote
0 answers
93 views

What is the difference between spliting the dataset into training and testing or collecting the training and testing data seperately?

I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. ...
Hamzah Al-Qadasi's user avatar
1 vote
0 answers
77 views

validation and test set definition for active learing with rare classes

Context I've got an active learning problem with an event rate of about 1%. The data is a panel, individuals over time. We have a proxy label that is highly correlated with the true label within ...
generic_user's user avatar
  • 13.4k
1 vote
0 answers
26 views

Experiment Design for Black-Box Function with Multiple Outputs

I have a multi-output black box function $f: x \rightarrow y$, where $x \in R^{M}$ and $y \in R^N$. Both $M$ and $N$ are greater than 1. For example, $M=4$, and $N=3$. My goal is to sample a set $\...
Hammer. Wang's user avatar
0 votes
0 answers
81 views

Is there a good framework for using Active Learning (reinforcement learning?) for Experimental Design?

We have a frequent problem that we deal with in a research environment that essentially boils down to finding the optimum conditions in a certain experimental space. At the moment we essentially solve ...
dsbbsd9's user avatar
1 vote
0 answers
34 views

Active Learning Bayes Learning

How was equation (1) derived. I am familiar with MAP but I cannot understand how this equation was derived using Bayes rule. It seems that in the Likelihood part we only consider the new training ...
zombiecs's user avatar
1 vote
0 answers
192 views

Active learning with Fisher information matrix for regression with vector output

I am trying to enhance the learning capabilities of my regression task with the help of active learning. I decided to use variance reduction using the Fisher information matrix as a query strategy as ...
Stacksatty's user avatar
1 vote
1 answer
197 views

The non i.i.d. problem resulted from active learning query strategy

We usually assume the i.i.d. assumption in machine learning problems, but in active learning, the labeled examples acquired by querying oracle are clearly not i.i.d. I want to know will it be better ...
Qcer's user avatar
  • 33
0 votes
0 answers
63 views

Implementing active learning in practice

I am working on a project where we are coding written survey responses pertaining to a persons mother tongue. For example if the person wrote in "English" it would get coded to 0001 and &...
astel's user avatar
  • 1,528
1 vote
1 answer
451 views

Is active learning with decision trees possible?

Background to make sure I understand decision trees: In order to create a decision tree, we go through each dimension and we attempt create two partitions of the data by trying every possible "...
bensenberner's user avatar
0 votes
1 answer
34 views

How to get boundary points that are at interface of differnet classes in multilabel classification dataset

I am working on finding points which are at boundary of different classes. In other words finding points on which a classifier would be most confused or uncertain about. For a setting like multi label ...
Nishant 's user avatar
2 votes
1 answer
52 views

Active learning with a unlabelled pool - standard references & model-based labelling of the pool?

I'm looking into active learning for a multi-class classification problem, where there is a large pool of unlabelled data. I start out with a small set of labelled data and can labelled some more of ...
Björn's user avatar
  • 33k
2 votes
1 answer
159 views

Different terminologies for ground truth

It's general theory that a supervised learning approach can only get as good an accuracy as the quality of the ground truth. The ground truth is considered the best possible annotation. However, ...
Christian's user avatar
  • 1,902
1 vote
0 answers
27 views

Active data acquisition with small datasets / Active Design of Experiment [closed]

Validated, I am facing a problem at work and I hope to find some light here, I have long worked with numerical methods, simulations and data analysis, but only recently I started studying and working ...
Jack of all Techs's user avatar
0 votes
1 answer
28 views

Notation in definition of a quantity involving uncertainty and posterior probability

As a probs & stats noob, I'm still getting confused by notation. I would appreciate if someone could elaborate a bit on what's going on here, on p12 of Settles' Active Learning survey (2012). He ...
Somatic Custard's user avatar
0 votes
1 answer
385 views

Model selection and error estimation with active learning

I'm working on a classification problem where we have plenty of unlabelled data but it is costly and time-consuming to label them. So, I'm looking into active learning to economize on how much data we ...
Milad Shahidi's user avatar
1 vote
0 answers
33 views

What is the convergence criteria of a semi-supervised learning algorithm

I would like to know about when to stop doing semi supervision? for example, I learn a classifier from a small dataset, and then I use it to label a pool of unlabelled dataset. How long should this ...
N. F.'s user avatar
  • 362
1 vote
0 answers
153 views

Active learning for object detection- Batch Selection

I have a small dataset of about 220 images for three classes. I am using YOLO (you only look once) network for an object detection. I am trying to use Active learning in order to reduce the number of ...
Desmanda's user avatar
2 votes
1 answer
677 views

Active Learning with Human-in-the-Loop

I did a lot of research and can't find a satisfactory answer. I have just a quick question about Active Learning and would be pleased if you could answer it. I'm still wondering if active learning ...
Gaudi33's user avatar
  • 21
7 votes
3 answers
235 views

Using ML to assist human labelling in dataset with highly unbalanced classes

Are there scientific issues with using ML to assist human annotation? I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. The labels arn't ...
Aidan Connelly's user avatar
3 votes
3 answers
2k views

Strategies for incorporating feedback for a ML algorithm

I am developing a text classification problem, in which at some time points, say at the end of each week, I receive a batch of feedback from users about correctly and wrongly classified inputs. I am ...
Hamed's user avatar
  • 31
1 vote
3 answers
115 views

Learning from Mistakes in a classification task

Let's say I have a text classifier that I have trained on some training dataset. Now when I run this trained classifier on some test dataset, I identify the cases where it went wrong (assuming I have ...
The Wanderer's user avatar
2 votes
1 answer
120 views

Are there any tutorial for text categorization with Active Learning?

I am going to categorize texts into predefined topics. It looks like Active Learning approach is suits for me. Are there any good tutorials or advices?
drobnbobn's user avatar
  • 457
0 votes
0 answers
44 views

Are there clustering algorithms which require a human as an oracle?

Are there data clustering algorithms which by definition require human judgement as part of the algorithm? What I mean is that once in a while the algorithm presents the human operator (or any other ...
Lior's user avatar
  • 557
5 votes
1 answer
145 views

Combine reinforces and unsupervised learning?

I have an existing set of data and plan to generate more data that follows the same pattern. To do this, I plan to use unsupervised learning. How can I provide feedback on the generated data and ...
Streetlamp's user avatar
1 vote
2 answers
785 views

Active learning using logistic regression

I've an unlimited amount of unlabelled data and an oracle that is able to label portions of this data using only two labels "1" or "0". I want to ask a few number of questions to the oracle and after ...
Enzo's user avatar
  • 11
2 votes
0 answers
97 views

Active learning system confusion matrix

A colleague of mine has been developing a machine learning system with an active learning component. I was having trouble reproducing the metrics he's been reporting, until I found out that he's ...
Richard Turner's user avatar
1 vote
0 answers
35 views

What description fits this this learning scenario, is this a form of active learning?

A system is involved in dialog with a human partner. The system has a model of a problem domain and knows mappings between words an concepts. The human has its own model of the problem domain and ...
samy's user avatar
  • 111
3 votes
1 answer
807 views

Batch Active Learning for classification?

Say we have unlimited unlabeled data and we can ask an oracle for labels. We can use active learning to choose the most informative data samples for labeling, thus minimizing data labeling cost. If we ...
Adam Kosiorek's user avatar
2 votes
0 answers
126 views

Active Learning Regression Problems [closed]

I Want to apply active learning in regression problems, but all the frameworks that I found are only for classification problems. Does someone know some framework for active learning in regression ...
Jorge Alberto Lopez's user avatar
1 vote
0 answers
131 views

Is a test set sampled after training examples have been removed by an active learning algorithm valid?

In this case desire is to optionally label a final test set after training. The training is to be done using an active learning approach which can be biased towards different characteristics (...
decostop's user avatar
4 votes
0 answers
1k views

R package for active learning and sampling [closed]

I tried to find R package for active learning/sampling and stumbled over "activelearning" package. However, this package is not available on CRAN. Does someone know how good is this package? Are ...
Michael's user avatar
  • 219
6 votes
1 answer
94 views

Labeling a pool of unlabelled samples iteratively

Problem setting I'm faced with a problem in which we have a large set of data points (100K), all of which are still unlabelled. These are to be used as input to a binary classifier at a later point ...
ciri's user avatar
  • 1,233
2 votes
1 answer
404 views

Asymmetrical selective sampling for linear classification

I've got a online classification problem where I predict a class label {+1, -1} for an object and then show it to a user to get a real label. My task is to minimize ...
martinthenext's user avatar