Questions tagged [corpus-linguistics]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

How was the perplexity of the Brown corpus measured?

I used the following code to calculate the perplexity of a corpus but it is giving me extremely low answer for bigram analyses. For Brown Corpus I am getting a result of 4.6. ...
0
votes
0answers
46 views

BERT Classification fine tuning for Q & A

I want to fine-tune BERT for Q & A in a different way than the SQuAD mission: I have pairs of (question, answer) Part of them are the correct answer (Label - 1) Part of them are the incorrect ...
0
votes
1answer
16 views

How to test normalized frequencies

I have extracted the frequency of two words (can, may) from two corpora (Corpus1, Corpus2). I want to test if there is a significant difference between the counts in the two corpora, but since the ...
0
votes
1answer
28 views

Why does model converge when factors are releveled?

I am building a mixed effects logistic regression to predict linguistic (corpus) data. I have coded for various factors, some of which do not correspond to a large amount of data. For example, one ...
0
votes
0answers
14 views

What are ways to build topic topic models on a corpus with high variance in text length

As the title states, I'm currently working with a corpus that has a high variance in the document length. Each input could be anything from the size of a tweet to a a full page worth of text. I'm ...
0
votes
1answer
25 views

Likelihood for a test data (sequence of characters) given two unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models ...
1
vote
0answers
25 views

How to compare multiple collocations based on their Chi2 score? How to find synonyms using collocation analysis

my question is regarding collocation analysis (i'm using R). I have following data gathered from Polish corpus: ...
1
vote
0answers
64 views

TF-IDF on a sub-corpus

I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long. My Corpus is made of a few thousands of dated articles spanning over a few ...
2
votes
1answer
42 views

Can a list of websites be considered a corpus for a particular category?

I am trying to build my own corpus for particular categories such as Engineering, Business, Math, Science and etc... This will be for automatic web page categorization. Let's say I manually collect ...
1
vote
1answer
32 views

How can I calculate if the difference between two word frequencies in one corpus is significant?

I want to study orthographical variants, for example: Can firefighter, fire-fighter and fire fighter be considered orthographical variants (the difference in frequency in a corpus is not ...
1
vote
0answers
50 views

Multiple comparisons problem in corpus linguistics

A collocation is an expression consisting of two (or more) words that correspond to some conventional way of saying things, for example strong - tea or new - car. We really want to know whether two ...