Consider the problem of semi-supervised learning where, in each round, the labels of all data points are guessed and then the label of a random data point is revealed. As the labels of more and more data points are revealed, the accuracy of the predicted labels should increase.

A plot comparing the accuracy of the predicted labels against the fraction of known labels (for several trials) might look like the following:

enter image description here

Our goal is to reach high accuracy as quickly as possible. Geometrically, we want our curve to rise as sharply as possible. I'm looking for a way to mathematically formalize this intuition, and I was thinking of using the total area under the curve.

Is this a good performance measure for semi-supervised learning in an iterative setting? If so, is it used anywhere in the literature?

$$\frac{1}{n} \sum_{i=0}^n \text{accuracy}_i \rightarrow \int_0^1 \text{accuracy} \, \mathrm{d}(\text{fraction queried})$$

where $n$ is the size of the data set (number of data points). For a method to be consistent we must have $\lim_{i \rightarrow n } \text{accuracy}_i = 1$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.