What kind of test shall I use to test if the two data sets are significantly different?

Question

The table below shows my example data set. I have 19 sentences (as an example). For each sentence, if word A is included it is recorded as 1; otherwise, it is recorded as 0 (shown in column A in the table below). Similarly, if word B is included in the sentence, it is recorded as 1; otherwise, it is recorded as 0 (shown in column B in the table below). I want to know if the appearance of word A is significantly different from word B. My question is, what kind of test shall I use? What I have thought is listed below. Thanks a lot in advance for your help!

1. I am thinking if my question is valid or not. When I start to look at the data set, it seems that the only information I can get from the data is how many times the word A and B appears. As the example shows, word A appears 8 times and B appears 5 times. If this is the only information we can get, that means we are comparing if 8 and 5 are significantly different. Then it seems not a valid question.

However, if we look at each sample, the data sets (A and B) are paired for each sentence. This is additional information other than the numbers (8 vs. 5). If this information (paired data) is meaningful, is my question is still valid? Do we have a statistical test that is suitable for this kind of question?

2. I have thought of using the t-test. But for each column (A and B), the distribution (only 0 and 1) is definitely not a $t$ distribution or normal distribution. So, I think the t-test is not suitable. Am I correct?

| #  | A | B |
|----|---|---|
| 1  | 1 | 1 |
| 2  | 0 | 0 |
| 3  | 1 | 0 |
| 4  | 0 | 0 |
| 5  | 1 | 0 |
| 6  | 1 | 0 |
| 7  | 1 | 0 |
| 8  | 0 | 0 |
| 9  | 0 | 1 |
| 10 | 0 | 1 |
| 11 | 0 | 0 |
| 12 | 0 | 0 |
| 13 | 1 | 1 |
| 14 | 0 | 1 |
| 15 | 1 | 0 |
| 16 | 0 | 0 |
| 17 | 0 | 0 |
| 18 | 1 | 0 |
| 19 | 0 | 0 |

Answer 1

If you're hypothesis is about the frequency of uses of the words in each of the 19 sentences, you can do a two sample test of proportions. There are LOTS of examples of that on this site, so I won't go through the details. If you have access to something like R, this is straightforward

library(tidyverse)

data = tribble(~'A', ~'B',
   1 , 1 ,
   0 , 0 ,
   1 , 0 ,
   0 , 0 ,
   1 , 0 ,
   1 , 0 ,
   1 , 0 ,
   0 , 0 ,
   0 , 1 ,
   0 , 1 ,
   0 , 0 ,
   0 , 0 ,
   1 , 1 ,
   0 , 1 ,
   1 , 0 ,
   0 , 0 ,
   0 , 0 ,
   1 , 0 ,
   0 , 0 )


appear_a = sum(data$A)
appear_b = sum(data$B)
prop.test(x = c(appear_a, appear_b), n = rep(nrow(data),2), correct = F )

The correct=F will make sure the result is consistent with the formulae you might find on most websites or textbook. The result is

    2-sample test for equality of proportions without continuity
    correction

data:  c(appear_a, appear_b) out of rep(nrow(data), 2)
X-squared = 1.0523, df = 1, p-value = 0.305
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.1395774  0.4553668
sample estimates:
   prop 1    prop 2 
0.4210526 0.2631579 

We fail to reject the null that the proportion of sentences that use each word is identical (p= 0.305)

You mention that it might be worth looking at the data as paired since they are from the same sentence. In that case, we should use a McNemar test to test the null hypothesis that the discordant frequency (that is, the probability that one word shows up but not the other) is the same for both words. Using your data

x = xtabs(~A+B, data = data)
mcnemar.test(x)
    McNemar's Chi-squared test with continuity correction

data:  x
McNemar's chi-squared = 0.44444, df = 1, p-value = 0.505

Again, we fail to reject the null.