Questions tagged [corpus-linguistics]
The corpus-linguistics tag has no usage guidance.
19
questions
3
votes
3
answers
488
views
Countering t-test "any feature is significant" results for large sample size datasets
I'm doing some analysis over natural language data, which basically entails:
Computing some feature over all samples.
Evaluating if this feature statistically significantly discriminates between ...
- 31
0
votes
0
answers
7
views
The right test for word usage in corpus linguistics
I need to find the right statistical test to use here:
the words on the left are prepositions.
the groups are beginner, Intermediate, and advanced level speakers of Swedish
My hypothesis is that ...
0
votes
1
answer
79
views
Estimating exponent of Zipf distribution using MLE vs fitting linear regression on log-transformed rank and frequency data
I'm having trouble understanding why I get radically different results if I try to find the parameter of a Zipf distribution when I use the methods proposed by Clauset et al. (2009) as opposed to ...
- 65
0
votes
0
answers
28
views
Distribution and difference between two categories in two corpora, and comparison
Building on this question:
I'd like to compute the difference between the presence of a 2 level factor in 2 datasets, and see whether this is significant.
For instance, let's say I want to find 2 ...
- 13
0
votes
0
answers
87
views
predicting the value of a nominal variable from the value of a ratio scale variable
I am doing a corpus study on the influence of subject length on word order choice between SVO and VSO orders. TO calculate that, I am using The Generalized Linear Model glm.
glm(formula = WordOrder ~ ...
Mohd Alamri
1
vote
1
answer
65
views
How do I tell if word frequencies are changing over time?
I have a collection of texts that span about 1000 years. I am interested in the frequency of a particular word in these texts. Specifically, I want to know whether the frequency of the word increased ...
- 409
1
vote
1
answer
189
views
What is the number of observations required for adequate Krippendorff's alpha?
say I am working on a project to classify a number of documents. Say I have 2,000 newspaper articles that I would like to code as either containing information of lawnmowers (1) or not (0). This is a ...
- 11
2
votes
1
answer
68
views
Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?
I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance.
I have not ...
- 23
0
votes
0
answers
186
views
BERT Classification fine tuning for Q & A
I want to fine-tune BERT for Q & A in a different way than the SQuAD mission:
I have pairs of (question, answer)
Part of them are the correct answer (Label - 1)
Part of them are the incorrect ...
2
votes
2
answers
511
views
Word perplexity on a subword language model
Let's have corpora
$X = x_1...x_N$
in which every word can be represented using subwords (from a fixed size vocabulary of subwords)
$x_i = x_{i,0}...x_{i,M(x_i)}$
where $M(x_i)$ is number of ...
- 31
0
votes
1
answer
100
views
How to test normalized frequencies
I have extracted the frequency of two words (can, may) from two corpora (Corpus1, Corpus2).
I want to test if there is a significant difference between the counts in the two corpora, but since the ...
0
votes
1
answer
285
views
Why does model converge when factors are releveled?
I am building a mixed effects logistic regression to predict linguistic (corpus) data. I have coded for various factors, some of which do not correspond to a large amount of data. For example, one ...
0
votes
1
answer
183
views
Likelihood for a test data (sequence of characters) given two unigram models
I would like to find the likelihood of a sequence of characters (the test data), given two unigram models.
The sequence (test data) is:
A B C B B
The models ...
- 101
1
vote
0
answers
101
views
How to compare multiple collocations based on their Chi2 score? How to find synonyms using collocation analysis
my question is regarding collocation analysis (i'm using R). I have following data gathered from Polish corpus:
...
- 11
1
vote
0
answers
92
views
TF-IDF on a sub-corpus
I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long.
My Corpus is made of a few thousands of dated articles spanning over a few ...
- 123
2
votes
1
answer
57
views
Can a list of websites be considered a corpus for a particular category?
I am trying to build my own corpus for particular categories such as Engineering, Business, Math, Science and etc... This will be for automatic web page categorization. Let's say I manually collect ...
user162992
1
vote
1
answer
49
views
How can I calculate if the difference between two word frequencies in one corpus is significant?
I want to study orthographical variants, for example:
Can firefighter, fire-fighter and fire fighter be considered orthographical variants (the difference in frequency in a corpus is not ...
2
votes
0
answers
157
views
Multiple comparisons problem in corpus linguistics
A collocation is an expression consisting of two (or more) words that correspond to some conventional way of saying things, for example strong - tea or new - car.
We really want to know whether two ...
- 2,213
2
votes
1
answer
313
views
How to know one system is significantly better than another one?
I am studying lexical semantics. I have 65 pairs of synonyms with their sense relatedness. The dataset is derived from the paper:
Rubenstein, Herbert, and John B. Goodenough. "Contextual correlates ...
- 121