# Advice on anomaly detection for a text

I have an idea of retreiving texts that have anomalous distribution of token frequency.

Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, some token appearance in texts will be correlated. E.g., 'music' and 'concert' and 'band'; 'oil' and 'price' and 'demand'. I can make quantitative measures of the pair-wise Pearson correlations, or, maybe, as a mutual information (or multiinformation).

On the next step I get a new text and I see that 'music' appears along with 'price', or 'oil' along with 'band'. This combination is Anomalous, but I can't see how to measure it.

$$A = f(p),$$ where p is a frequency distribution of tokens in a text.

And it is important that p can be quite a long vector (50,000 tokens in total or so), and the anomaly may happen in just a little part of it.

Q: Could you give a hint on either method, idea, paper, or a concept how to go with this?

UPDATE.

After thinking more on the problem I came to:

$$p(A|B) = \frac{P(A) * P(B|A)} {P(B)},$$

where A and B are tokens.

I guess what I need is to detect cases where p is small for pairs of tokens.

But I need to calculate this for all pairs of A and B. And take some sort of a statistic on this matrix. Any ideas would be helpful.

UPDATE

I made some research and coding. Below is a proposed function of anomaly score for a text. text is a named vector of token frequencies.

I take all pairs of tokens from it, calculate a Bayesian posterior, then I have a square matrix (zeroed diagonal) and as a first try I get median value of probabilities. It should read as "in this text median pairwise conditional probability is low enough to suppose anomalous text content..." But I am still not sure it is sound in terms of math or any theorem.

## calculate bayesian stats ----------

require(data.table)

bayes_anomaly <-
function(
text
)
{

st <-
lapply(
names(text)
, function(x)
{

A <- x

Bs <- numeric()

for(
B in names(text)
)
{

p_A <- mean(DTM[, A, with = F][] > 0)

p_B <- mean(DTM[, B, with = F][] > 0)

dt_BA <-
DTM[, c(A, B), with = F]

colnames(dt_BA) <- c('A', 'B')

p_BA <- dt_BA[A > 0, mean(B > 0)]

p_AB <- (p_A * p_BA) / p_B

names(p_AB) <- B

Bs <- c(Bs, p_AB)

}

Bs

}

)

dt_st <-
do.call(rbind, st)

diag(dt_st) <- 0

rownames(dt_st) <- names(text)

median(dt_st)

}