0
$\begingroup$

I have the following requirement: I get a list of words. These words come from table-like structures (they do not have table column headers). They are not well-formed sentences. Example input:

['th', 'March, 2018', 'LAST', 'COMMENT', '06-Mar', 'Gibraltar', 'Sea Helios', 
 '45,948', '52,719', 'N/A', 'UMS']

From the above list, I have to tag some words to their respective category. For example:

{
    '06-Mar': 'date',
    'Gibraltar': 'location',
    'Sea Helios': 'name',
    '45,948': 'weight'
}

In the input list, not every word may belong to a category. Some are just rubbish words. I have been looking at Word2Vec techniques, however, they all depend on sentences for training. Can this be done with classification/clustering? Any pointers on how I may be able to do this? The date formats are variable like 06-Mar, 06-07/Mar, etc. Other entities like name and location may be misspelled so a direct lookup from a hashmap is not possible.

Improve this question
$\endgroup$
4
  • $\begingroup$ Detecting elements like dates or weight in your example seem more like a problem to be tackled with regular expressions rather than statistical methods. While your 'location' category could be matched against a dictionairy of locations, I find your definition of 'name' somewhat vague. Can you elaborate a bit more on that? $\endgroup$
    – deemel
    Commented Mar 19, 2018 at 13:22
  • $\begingroup$ @Rickyfox The way a human knows that it's a name is by domain knowledge. Basically, they are vessel names and people in industry sort of knows that it is. Having said that, I have access to a database with names and if I use string similarity I can get a very accurate result. Just that it can be slow because this is expected to work with large data sets. Same goes with regex. Regex is also my backup plan if I can't use a statistical technique. $\endgroup$
    – kovac
    Commented Mar 19, 2018 at 13:33
  • $\begingroup$ Well that makes it a bit clearer. However, I'm not aware of a statistical procedure that you can use with this that does not require some notion of string similarity, especially given that the text variables can be potentially misspelled. $\endgroup$
    – deemel
    Commented Mar 19, 2018 at 13:44
  • $\begingroup$ Thank you for your advice. I have also been checking for a few days now but all the apis out there assumes sentences. $\endgroup$
    – kovac
    Commented Mar 19, 2018 at 13:45

1 Answer 1

Reset to default
0
$\begingroup$

If I understood the question right, your task is more Information Retrieval than Machine Learning, So word2vec does not help you. What you are looking for is called Knowledge-Base. But why?

First of all, the knowledge stored in the system should be as huge as possible as you can imagine "Gibraltar" is not the most famous place. So something in the scale of web works e.g. Wikipedia (if it is a place most probably it has a Wikipedia page).

Second of all this data should be in the form of an Ontology which stores the relation between concepts and answer the question "What is X?" e.g. Wikipedia is an ontology which tells you Lionel Messi is an Argentinian football player, Lionel Messi is a player of Barcelona, etc.

Third of all this information should be structured in a graph as you need to relate every entity and it's category (parent) e.g. like how Wikipedia links the category of its pages in "Categories" section at the bottom of the page. Ontologies are usually stored in knowledge graphs (knowledge graph as a general term. do not confuse by specific knowledge-base used by google).

Solution

After this introduction I can simply give you some keywords to search. Then you will easily find the best practices for answering your question:

Semantic Web, WordNet, DBpedia, Google Knowledge Graph, RDF

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.