For our ML assignment we have three datasets. The challenge is about checking whether a written and spoken number refer to the same number. We're using the MNIST dataset with handwritten numbers, and an audio dataset containing Arabic spoken numbers. Our data looks like this:
Written train dataset.npy (45.000 rows, 784 columns with pixel values) Spoken train dataset.npy (45.000 rows, each row is (N, 13)) Match_train dataset.npy (45.000 rows with boolean values, True or False)
The match dataset refers to a True when the written and spoken data refer to the same number.
We thought about labeling the written dataset and comparing them to an labeled Arabic set, but then we have no labels for the Arabic spoken numbers. We don't see a way to label the spoken numbers. We thought about using the Match_train dataset with Boolean values as labels, but we don't know how to interpret this exactly.
How would you guys tackle this problem?