How many emails would I need to train a good text extraction model?

Question

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly label only products and not anything else. Just to make things crystal clear, these types of items would ideally be labeled as products:

• SAMSUNG: EVO Select 128GB MicroSDXC UHS-I U3 100MB/s Full HD & 4K UHD Memory Card with Adapter (MB-ME128HA)
• WAYF x BFF Hollie Long Sleeve Sweater Dress
• Super Mario 3D All-Stars - Nintendo Switch, Nintendo Switch Lite

and here are some non-examples:

• Kansas City Chiefs (unless it was associated with jerseys a customer bought)
• New York, NY 10025
• Python programming language

I'm new to the field of NLP and would like to know how many emails I should look at to start off with to train an initial model. Also, what type of model(s) should I try out first? Any help in either of those questions would be greatly appreciated.

Answer 1

I haven't done this exact problem, but I can take a stab in the absence of better answers... It looks like this paper is pretty close to your problem, and this one is a similar problem in a different industry. The first will give you lots of ideas about techniques, but has hundreds of thousands of text samples, plus a list of millions of products. They only had "hundreds" of labeled text samples, but it seems even the best models were pretty bad. You might poke around and see if you can find that list of products they used.

The second paper has learning curves, which helps get at your first question. It seems like they needed tens of thousands of words before performance got good. But this is likely a bare minimum, because they were using Active Learning, which cuts down on samples needed dramatically (something you could also explore).

So I'd think you'd want thousands, if not tens of thousands of LABELED emails. Which brings up an important question: will your emails be labeled? Do you have a mechanism by which humans will manually identify which emails have products and which do not? This is a ton of manual work, but will make the ML problem much easier.

If you do not have labels, but just a random collection of emails, then you are talking about unsupervised learning, which complicates things and will take more creativity. You might explore publicly available models that were trained on labeled data that you could adapt to your problem, possibly through transfer learning.