1
$\begingroup$

I have a dataset that has probability labels instead of one-hot-encoded labels AND I'd like to keep it that way. The probabilities of each sample is information that I'd like to influence the loss function during backprop.

E.g. of dataset

Sample (Movie) Horror Comedy Drama SciFy/Fantasy
Aliens 0.3 0.2 0.0 0.5
Dumb & Dumber 0.0 0.8 0.2 0.0
The Hobbit 0.1 0.2 0.2 0.5

Above you can see each movie is labeled corresponding to a category with a probability in each category. One-hot-encoding would hard classify Aliens as 100% SciFy/Fantasy, but there's value in knowing that this also belongs to horror and comedy too and in those exact proportions.

My usecase is a little more complicated than this as well. I'm not able to simply slap on scikit-learn's logistic regression model and call it a day. I'll need to use a timeseries neural network. I'm able to get decent results with my current model that converts everything to a one-hot-encoded classification, but I believe that it would perform significantly better if y_target utilized more loose class labels reflecting the probabilities in each of those labels.

I'm assuming that using those y_target probabilities will work just like any one-hot classification problem and that the loss function will reflect these softer labels, but I am unsure...

What I'm thinking is:

linear model output --> y_pred --> softmax << cross-entropy >> y_target

Essentially, use a linear output from the model. Then squash it down using softmax. Then calculate the loss with cross-entropy against y_target. y_pred and y_target will both have probability distributions and both will not be one-hot encoded.

Can this be done? and is it doing what I think it's doing? Has anyone seen any good examples of something similar?

Improve this question
$\endgroup$
3
  • $\begingroup$ Interesting question and a +1 from me, but I am concerned that you might be mixing up two ideas. It seems like your data are saying that Aliens is $30\%$ horror, $20\%$ comedy, and $50\%$ sci-fi/fantasy. What does that have to do with weights in backprop for a softmax neural network regression? // In a standard multinomial logistic regression, the output contains the probabilities of each class. This can be used to make a hard classification, but it does not have to be (and there are advantages to using the probability values instead of hard classifications). $\endgroup$
    – Dave
    Dec 23, 2021 at 0:14
  • $\begingroup$ This is obviously not something I've seen anywhere... I've dreamed up this approach. The reason why I think it'll impact cross-entropy loss is due to its formula. H(P, Q) = – sum x in X P(x) * log(Q(x)) The probability distribution P(x) is multiplied by Q(x) for each class/label. So, in a typical 1-hot function, only one of these labels is ultimately calculated. My proposed approach would actually use the sum of the difference between prob distributions... $\endgroup$
    – user16796559
    Dec 23, 2021 at 1:47
  • $\begingroup$ Focal Loss looks very promising. It's essentially cross-entropy with a weighting criteria. arxiv.org/pdf/1708.02002.pdf Perhaps I can use hard 1-hot encoded y_targets, but use those percentages/probabilities as the weighting... $\endgroup$
    – user16796559
    Dec 23, 2021 at 16:26

1 Answer 1

Reset to default
0
$\begingroup$

Answering my own question. Yes, you can use cross entropy loss for comparing two vectors of continuous values. It's stated in the wiki with supporting proofs:

The situation for continuous distributions is analogous.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.