I am working on a project where we are coding written survey responses pertaining to a persons mother tongue. For example if the person wrote in "English" it would get coded to 0001 and "Spanish" would get coded to 0002 etc. To do this we have created a reference file that will catch everything we expect to see. For instance the reference file will have English and Spanish etc.

The issue is we have potentially millions of responses written in that may not match to the reference file. For example, spelling mistakes or maybe colloquially written terms, sometimes just nonsense is written in etc. We would like to use machine learning to process these write-ins that the reference file does not catch. The problem is that we do not have "true" values from which to train from beyond the reference file.

We could try using the reference file as a training set but the performance will likely be poor. We do have "experts" that can look at a write-in and assign the correct code so I was wondering if we could build an initial model from the reference file and use active learning to improve it in the following way.

  1. Build initial model on the reference file
  2. Select two samples from records not matching reference file; first select a simple random sample from the population to be used to analyze performance and the second sample selected from records that were particularly difficult for the model to predict (i.e. those with equal probabilities between classes).
  3. Have our expert code both samples
  4. Calculate performance on the first sample
  5. Retrain model with reference file data plus data from both samples
  6. Repeat 1-5 until performance stops increasing significantly

Does this approach sound valid? Am I leaking data somehow by doing this? Is there a better approach?

  • 1
    $\begingroup$ A good starting point would be to apply a pertained language detection model, and see how it performs a large sample of your surveys. If you're worried about spelling mistakes, a character-level (or byte-pair) classifier should work decently well. Looking at the false positives will elucidate which areas need improvement. Shorter sentences and languages that mix in English will likely be difficult examples. $\endgroup$
    – Alex R.
    Oct 7 '20 at 19:43
  • $\begingroup$ The entire problem is the data isn’t labeled. I have no way to measure false positives or any other performance metrics for that matter... $\endgroup$
    – astel
    Oct 7 '20 at 20:02
  • $\begingroup$ SRS=Random sampling? $\endgroup$
    – Saleh
    Oct 8 '20 at 5:38
  • $\begingroup$ @astel 1) You need labeled data to communicate to the machine what it is that you want. You can use your reference to label some data automatically, and also label some bad examples by hand. Most deep learning software will give you a confidence for each tested datapoint - after training you can evaluate the classifier on some unlabelled data, select those for which the confidence is lowest. Then label some of those by hand, re-train, continue until fraction of unconfident classifications is satisfactory $\endgroup$ Oct 12 '20 at 12:39
  • $\begingroup$ @astel 2) So what exactly is the problem with simply counting the number of words in each response which match to a given dictionary? Even if a few words are misspelled, isn't there still a sweet spot threshold on the fraction of found words that would allow to perform dictionary-based classification? How many words are there in each response? $\endgroup$ Oct 12 '20 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.