I have an idea of retreiving texts that have anomalous distribution of token frequency.

Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, some token appearance in texts will be correlated. E.g., 'music' and 'concert' and 'band'; 'oil' and 'price' and 'demand'. I can make quantitative measures of the pair-wise Pearson correlations, or, maybe, as a mutual information (or multiinformation).

On the next step I get a new text and I see that 'music' appears along with 'price', or 'oil' along with 'band'. This combination is Anomalous, but I can't see how to measure it.

$$ A = f(p), $$ where p is a frequency distribution of tokens in a text.

And it is important that p can be quite a long vector (50,000 tokens in total or so), and the anomaly may happen in just a little part of it.

Q: Could you give a hint on either method, idea, paper, or a concept how to go with this?


After thinking more on the problem I came to:

$$ p(A|B) = \frac{P(A) * P(B|A)} {P(B)}, $$

where A and B are tokens.

I guess what I need is to detect cases where p is small for pairs of tokens.

But I need to calculate this for all pairs of A and B. And take some sort of a statistic on this matrix. Any ideas would be helpful.


I made some research and coding. Below is a proposed function of anomaly score for a text. text is a named vector of token frequencies.

I take all pairs of tokens from it, calculate a Bayesian posterior, then I have a square matrix (zeroed diagonal) and as a first try I get median value of probabilities. It should read as "in this text median pairwise conditional probability is low enough to suppose anomalous text content..." But I am still not sure it is sound in terms of math or any theorem.

## calculate bayesian stats ----------


bayes_anomaly <- 

    st <-
        , function(x)

          A <- x

          Bs <- numeric()

            B in names(text)

            p_A <- mean(DTM[, A, with = F][[1]] > 0)

            p_B <- mean(DTM[, B, with = F][[1]] > 0)

            dt_BA <- 
              DTM[, c(A, B), with = F]

            colnames(dt_BA) <- c('A', 'B')

            p_BA <- dt_BA[A > 0, mean(B > 0)]

            p_AB <- (p_A * p_BA) / p_B

            names(p_AB) <- B

            Bs <- c(Bs, p_AB)





    dt_st <- 
      do.call(rbind, st)

    diag(dt_st) <- 0

    rownames(dt_st) <- names(text)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.