I would like to know how to deal with data that misses a label, but is likely to contain the label in a weakly supervised setting.

Weakly supervised background

Since labeling is a time consuming and noisy process (e.g. Should a timestamp of an event be at 5.4 or 5.5 seconds?), some people instead take a weakly labelled approach. This approach only provides a label for the whole recording. E.g. an audio file contains events x, y, z without any temporal information for each time stamp. A database containing weak labels is the YouTube-8M database.

The first 2 chapters of the paper Data-Efficient Weakly Supervised Learning for Low-Resource Audio Event Detection Using Deep Learning does a good job describing the issue.

Problem description

I'm turning Mozilla's Common Voice dataset into a weakly supervised dataset. I do this by concatenating audio files (4+-3 seconds) into 10 second files. Then as output label for the whole 10 sec audio file I say it contains e.g. the accent labels ["eng_us", "eng_england", "eng_australia"]. If I want to recognize 5 English accents, I would e.g. encode this into [1 0 1 1 0] (with the other 2 being e.g. ["eng_indian", "eng_scotland"].

Due to user's not always specifying their information, such as "gender", "accent" and such, the data might miss the label "eng_england", while it is clearly in the voice data. Unlike the YouTube-8M, where I assume that a video containing e.g. an explosion, but not labeled as such is rather small, in the common voice dataset it is a high amount. E.g. "Gender" has 55.029 "male", 18.249 "female", 781 "other", 121.717 "NaN".


  • What is the best way to encode the labels for these concatenated audio files?
  • If a value is NaN, would it be better to take a soft labeling approach i.e. [0.2 0.2 0.2 0.2 0.2]?
  • Would a concatenated file with labels [0 0 1 1 0] give a problem, while it should actually have been [1 0 1 1 0], because person A didn't say he/she has an "eng_us" accent?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.