I am trying to build my own corpus for particular categories such as Engineering, Business, Math, Science and etc... This will be for automatic web page categorization. Let's say I manually collect 100 websites that are related to Math. Can these 100 websites be considered a corpus for Math?

Another related question. How does this differentiate from a lexicon wherein instead of a list of websites it shows a list of words with weights such as 0 or 1 to particular categories? Example would be a sentiment lexicon with words that has weights for positive and negative. But instead of positive and negative, categories such as Math, Science are used.

  • $\begingroup$ i really think so. $\endgroup$ – el Josso Jul 25 '17 at 9:58
  • $\begingroup$ @elJosso What would the difference be then from a lexicon? I have edited the question. $\endgroup$ – John Jul 25 '17 at 10:46
  • $\begingroup$ my answer is not satisfying ? $\endgroup$ – el Josso Aug 7 '17 at 12:18

There may be a misunderstanding about Lexicon. Let me paraphrase Hady Elsahar, Master's Degree holder in NLP and computational linguistics

Corpus : "A corpus is a large body of natural language text used for accumulating statistics on natural language text. The plural is corpora. Corpora often include extra information such as a tag for each word indicating its part-of-speech, and perhaps the parse tree for each sentence. "

Lexicon : "A lexicon is a collection of information about the words of a language about the lexical categories to which they belong. A lexicon is usually structured as a collection of lexical entries, like ("pig" N V ADJ). "pig" is familiar as a N, but also occurs as a verb ("Jane pigged herself on pizza") and an adjective, in the phrase "pig iron", for example. In practice, a lexical entry will include further information about the roles the word plays, such as feature information - for example, whether a verb is transitive, intransitive, ditransitive, etc., what form the verb takes (e.g. present participle, or past tense, etc.) "

I think everything is in his answer, here you will have a large body of text about math. And you will be able to know wich word refers to wich field.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.