Skip to main content
removed thanks
Source Link
Jan Kukacka
  • 11.5k
  • 1
  • 44
  • 70

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in classifying words by semantics and extracting meaning.

Right now I'm looking for a way to do something a little more difficult. Suppose that you have a text with plain text and multiple numerical codes in it. Some of these codes can be filtered by ruling, as they have common type characteristics (for example, a date or a telephone number from a specific country). So it should be very easy to classify those numbers by their meaning.

But some other numbers have almost no ruling. Among them we have a user identifier code which can be chosen by the user and can contain numbers, letters or other symbols, which lenghts can vary greatly. The user could provide their information on a written plain text file in a natural (or rather chaotic) way such as this:

Name: Mike
Home address: 234 Port Street
Work addr: Park Lane 14
555-6758
U. code: 334-5
2019/05/01
28 years old

Another participant could send us a text file such as this:

Veronica Phillips
I live in Barcelona, Diagonal Avenue, 6.
Joined 06-23-2019
Phone numer: 973 45 46 47
Personal identity: 87!3
I was associated with class 1A science team when in High School.

So the information provided would require some data to be present (address, name, date joined, user chosen code), and would allow for optional data such as the age or other records. It also would allow poor structure and typos (like in numer insted of number on the last example).

Let's provide one more:

Name:    Addr:          Phone:     Usercode:   Joining date:   More data:
Mark S.  Baker St., 231 987 65 43  777-7.3A     2017-10-10     6.5 feet tall.
         London

As chaotic as it can be (and I'm doing this on purpose for a challenge), a human viewer would easily be able to extract critical information about the user and be able to store their names, addresses, user codes and so on.

Yet here we don't have clear rules for all the strings we want to identify, sometimes code words to identify them won't be just at their side, could be missing, far away (last example) or worded in another way (Usercode / User number / My entry code).

What would be the best strategy to tackle something like this problem using Machine Learning? Is there a good solution for this yet?

Thank you a lot for your time and aswers.

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in classifying words by semantics and extracting meaning.

Right now I'm looking for a way to do something a little more difficult. Suppose that you have a text with plain text and multiple numerical codes in it. Some of these codes can be filtered by ruling, as they have common type characteristics (for example, a date or a telephone number from a specific country). So it should be very easy to classify those numbers by their meaning.

But some other numbers have almost no ruling. Among them we have a user identifier code which can be chosen by the user and can contain numbers, letters or other symbols, which lenghts can vary greatly. The user could provide their information on a written plain text file in a natural (or rather chaotic) way such as this:

Name: Mike
Home address: 234 Port Street
Work addr: Park Lane 14
555-6758
U. code: 334-5
2019/05/01
28 years old

Another participant could send us a text file such as this:

Veronica Phillips
I live in Barcelona, Diagonal Avenue, 6.
Joined 06-23-2019
Phone numer: 973 45 46 47
Personal identity: 87!3
I was associated with class 1A science team when in High School.

So the information provided would require some data to be present (address, name, date joined, user chosen code), and would allow for optional data such as the age or other records. It also would allow poor structure and typos (like in numer insted of number on the last example).

Let's provide one more:

Name:    Addr:          Phone:     Usercode:   Joining date:   More data:
Mark S.  Baker St., 231 987 65 43  777-7.3A     2017-10-10     6.5 feet tall.
         London

As chaotic as it can be (and I'm doing this on purpose for a challenge), a human viewer would easily be able to extract critical information about the user and be able to store their names, addresses, user codes and so on.

Yet here we don't have clear rules for all the strings we want to identify, sometimes code words to identify them won't be just at their side, could be missing, far away (last example) or worded in another way (Usercode / User number / My entry code).

What would be the best strategy to tackle something like this problem using Machine Learning? Is there a good solution for this yet?

Thank you a lot for your time and aswers.

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in classifying words by semantics and extracting meaning.

Right now I'm looking for a way to do something a little more difficult. Suppose that you have a text with plain text and multiple numerical codes in it. Some of these codes can be filtered by ruling, as they have common type characteristics (for example, a date or a telephone number from a specific country). So it should be very easy to classify those numbers by their meaning.

But some other numbers have almost no ruling. Among them we have a user identifier code which can be chosen by the user and can contain numbers, letters or other symbols, which lenghts can vary greatly. The user could provide their information on a written plain text file in a natural (or rather chaotic) way such as this:

Name: Mike
Home address: 234 Port Street
Work addr: Park Lane 14
555-6758
U. code: 334-5
2019/05/01
28 years old

Another participant could send us a text file such as this:

Veronica Phillips
I live in Barcelona, Diagonal Avenue, 6.
Joined 06-23-2019
Phone numer: 973 45 46 47
Personal identity: 87!3
I was associated with class 1A science team when in High School.

So the information provided would require some data to be present (address, name, date joined, user chosen code), and would allow for optional data such as the age or other records. It also would allow poor structure and typos (like in numer insted of number on the last example).

Let's provide one more:

Name:    Addr:          Phone:     Usercode:   Joining date:   More data:
Mark S.  Baker St., 231 987 65 43  777-7.3A     2017-10-10     6.5 feet tall.
         London

As chaotic as it can be (and I'm doing this on purpose for a challenge), a human viewer would easily be able to extract critical information about the user and be able to store their names, addresses, user codes and so on.

Yet here we don't have clear rules for all the strings we want to identify, sometimes code words to identify them won't be just at their side, could be missing, far away (last example) or worded in another way (Usercode / User number / My entry code).

What would be the best strategy to tackle something like this problem using Machine Learning? Is there a good solution for this yet?

Name: Mike Home address: 234 Port Street Work addr: Park Lane 14 555-6758 U. code: 334-5 2019/05/01 28 years old

Name: Mike
Home address: 234 Port Street
Work addr: Park Lane 14
555-6758
U. code: 334-5
2019/05/01
28 years old

Veronica Phillips I live in Barcelona, Diagonal Avenue, 6. Joined 06-23-2019 Phone numer: 973 45 46 47 Personal identity: 87!3 I was associated with class 1A science team when in High School.

Veronica Phillips
I live in Barcelona, Diagonal Avenue, 6.
Joined 06-23-2019
Phone numer: 973 45 46 47
Personal identity: 87!3
I was associated with class 1A science team when in High School.

Name: Addr: Phone: Usercode: Joining date: More data: Mark S. Baker St., 231 987 65 43 777-7.3A 2017-10-10 6.5 feet tall. London

Name:    Addr:          Phone:     Usercode:   Joining date:   More data:
Mark S.  Baker St., 231 987 65 43  777-7.3A     2017-10-10     6.5 feet tall.
         London

Name: Mike Home address: 234 Port Street Work addr: Park Lane 14 555-6758 U. code: 334-5 2019/05/01 28 years old

Veronica Phillips I live in Barcelona, Diagonal Avenue, 6. Joined 06-23-2019 Phone numer: 973 45 46 47 Personal identity: 87!3 I was associated with class 1A science team when in High School.

Name: Addr: Phone: Usercode: Joining date: More data: Mark S. Baker St., 231 987 65 43 777-7.3A 2017-10-10 6.5 feet tall. London

Name: Mike
Home address: 234 Port Street
Work addr: Park Lane 14
555-6758
U. code: 334-5
2019/05/01
28 years old
Veronica Phillips
I live in Barcelona, Diagonal Avenue, 6.
Joined 06-23-2019
Phone numer: 973 45 46 47
Personal identity: 87!3
I was associated with class 1A science team when in High School.
Name:    Addr:          Phone:     Usercode:   Joining date:   More data:
Mark S.  Baker St., 231 987 65 43  777-7.3A     2017-10-10     6.5 feet tall.
         London
Source Link

Finding a varying code into a text

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in classifying words by semantics and extracting meaning.

Right now I'm looking for a way to do something a little more difficult. Suppose that you have a text with plain text and multiple numerical codes in it. Some of these codes can be filtered by ruling, as they have common type characteristics (for example, a date or a telephone number from a specific country). So it should be very easy to classify those numbers by their meaning.

But some other numbers have almost no ruling. Among them we have a user identifier code which can be chosen by the user and can contain numbers, letters or other symbols, which lenghts can vary greatly. The user could provide their information on a written plain text file in a natural (or rather chaotic) way such as this:

Name: Mike Home address: 234 Port Street Work addr: Park Lane 14 555-6758 U. code: 334-5 2019/05/01 28 years old

Another participant could send us a text file such as this:

Veronica Phillips I live in Barcelona, Diagonal Avenue, 6. Joined 06-23-2019 Phone numer: 973 45 46 47 Personal identity: 87!3 I was associated with class 1A science team when in High School.

So the information provided would require some data to be present (address, name, date joined, user chosen code), and would allow for optional data such as the age or other records. It also would allow poor structure and typos (like in numer insted of number on the last example).

Let's provide one more:

Name: Addr: Phone: Usercode: Joining date: More data: Mark S. Baker St., 231 987 65 43 777-7.3A 2017-10-10 6.5 feet tall. London

As chaotic as it can be (and I'm doing this on purpose for a challenge), a human viewer would easily be able to extract critical information about the user and be able to store their names, addresses, user codes and so on.

Yet here we don't have clear rules for all the strings we want to identify, sometimes code words to identify them won't be just at their side, could be missing, far away (last example) or worded in another way (Usercode / User number / My entry code).

What would be the best strategy to tackle something like this problem using Machine Learning? Is there a good solution for this yet?

Thank you a lot for your time and aswers.