11
$\begingroup$

Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:

+----------+----------+-------+-----------+
| FeatureA | FeatureB | Label | Certainty |
+----------+----------+-------+-----------+
|        2 |        3 | A     | 50%       |
|        3 |        1 | B     | 80%       |
|        1 |        1 | A     | 100%      |
+----------+----------+-------+-----------+

How could we train a machine learning model on this? Thanks.

$\endgroup$
11
$\begingroup$

As a numerical quality you ascribe to your data, I think this "certainty" could surely be used as a weight. Higher "certainty" scores increase the weight a datum has on the decision function, which makes sense.

Many supervised learning algorithms support weights, so you just have to find a weighted version of the one you intend to use.

$\endgroup$
  • 2
    $\begingroup$ (+1) And since essentially the weights will tend to act as "replicates" of points, probably any algorithm could be kludged into a weighted version that way, e.g. in the OP example, pass in [5,8,10] copies of the 3 points, reflecting their certainties of [50,80,100]%. (This should never be truly needed, as if it could be done in principle, there should be a corresponding weighted version of the algorithm.) $\endgroup$ – GeoMatt22 Jan 7 '17 at 4:33
5
$\begingroup$

Instead of having labels A or B, you could replace them with continuous values of the certainty -- for example, $1$ corresponds to something you're sure is $A$, $0$ corresponds to something you're sure is $B$ and $0.6$ corresponds to something you're 40% sure is $A$. Then, have a model that instead of predicting class $A$ or $B$ outputs a score between $0$ and $1$ based on how much you think its one or the other (and threshold this score based on if its > or < 1/2). This turns your classification problem into a regression problem (which you threshold to get back to a classifier).

For example, you could fit a linear model to $\log \frac{p(A|x)}{p(B|x)} = \log \frac{p(A|x)}{1-P(A|x)} $ as $ \beta_0 + \beta_1^T x $ (where $p(A|x)$ is the certainty above). Then, when you want to test some data, plug it into the model, and output label $A$ if $ \beta_0 + \beta_1^T x >0 $ and $B$ otherwise.

$\endgroup$
  • $\begingroup$ So if you had a multi-class classification problem, you could set your targets as vectors with length equal to the number of classes? $\endgroup$ – hyperdo Jan 7 '17 at 4:00
  • $\begingroup$ Number of classes -1, assuming the certainties sum to 100%; the example is similar to logistic regression. A lot of classifiers produce scores (e.g. estimates of the p(class | data) under some model). All this answer proposes is that instead of predicting the classes directly, view the certainties as scores, and predict them instead. Then, do something with the scores. $\endgroup$ – Batman Jan 7 '17 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.