1
$\begingroup$

Problem Background: I have free form text data with more or less arbitrary formatting/structure, but semantically it can be broken down like this:

{header}
{title}
{subtitle}
{article}
{boilerplate}

However, not all sections need necessarily have Length > 0, except {article} and {title}. Another thing to consider is that by far the longest section is {article}, often $\gtrapprox 90\%$ of the whole text. In the end I want to find those sections within a given text.

The {header} is not just a line or two, it often is several lines of text unrelated to the other sections, and is usually a couple of sentences addressing the following {article} and what should be done with it. In other words the {header} is often a full email, which precedes the {article}. All remaining sections are related to the {article} section.

Example Data: An example might look like this:

Hello Mr. X,

I've attached a new press release regarding Y and would like for you to do Z with it.

If you have any questions I would be glad to answer them.
Best Regards,
Mr. X

Awesome News
Really awesome news about some stuff.

A long and nicely formatted couple of paragraphs with pictures tables and
whatnot follows...

To learn more visit some website
Or contact Mr. X

Attempt: I think trying to label individual words according to the section they are in won't do me much good, not without training some deep network with millions of samples, which I don't have.

I've thought about other approaches. Such as focusing purely on the structure, e.g.:

  • Line Length
  • #words in line
  • Capitalization
  • Relative Position of Line in Text
  • Number of paragraphs
  • ...

i.e. all kinds of structure I can find in the text. This approach while, presumably, not as much as a purely word based approach would still result in a lot of bias towards the {article} section.

Another thought I had was to transform the text labeling into an image segmentation problem. Since, most of the time, the sections could be easily divided with four horizontal lines if converted into a picture, and the font type/size plus other formatting and position indicators would also help.

Question: Could you maybe point me in a possible direction I could take to tackle the problem ?

What would be a good model ?

Is there maybe some paper which discusses text part segmentation as I want to do it and how to avoid bias ?

Is the problem maybe not a good fit to be solved through a ML approach ?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.