Language Detection with CLD2 with Mixed Inputs in long documents

Question

Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages.

When working on long documents size like

~3000-4000 words, 
~40-50.000 characters

of mixed input texts (at least 2-3 languages in the same document), I see that CLD fails the recognize all the mixed inputs, resulting in only the most common language, like being having a polarization around this language like in this document excerpt:

Only come and treat me right
And you'll never guilty
Sekarang kamu sudah ada di depanku
Aku pun berdebar menanti kata-katamu
Honey Bunny Sweety
Let's take a chance

This will be recognized as english so I get

{
  "results": [
    {
      "reliable": true,
      "detection": {
        "name": "ENGLISH",
        "code": "en",
        "percent": 54,
        "score": 930
      }
    }
  ]
}

while I would expect here to have at least 2 languages. Internally CLD2 uses NGram decomposition of the input text, that is known to perform very will on language detection in a text. See here for more details.

If I generate ngram of a given size (this case N=2) of this document I will get this time

{
    "count": 86,
    "code": "na",
    "name": "na",
    "mean": 0.45989304812834225
  },
  {
    "count": 50,
    "mean": 16.503352692086242,
    "code": "id",
    "name": "INDONESIAN"
  },
  {
    "count": 38,
    "mean": 12.779225483523962,
    "code": "en",
    "name": "ENGLISH"
  },
  {
    "count": 13,
    "mean": 1.5371176291771826,
    "code": "ms",
    "name": "MALAY"
  }

i.e. a more detailed detection of the mixed input language that are in this document. Of course this detection depends on N i.e. the size of the ngrams, so it may happens that for some values of N it have false positive (like a new language detected that it is not in the mixed inputs).

Assumed that CLD2 is using Ngram internally, and that the Bayes classifier was trained on ~200 characters wide text (~2-3 sentences), it seems to have a polarization in some way, and to provide better results, when working in this way - in the case of mixed inputs. The question is if this is arguable in some way and if there is a different approach that could bring to the same results obtained here.

Answer 1

Edit: My answer is wrong, read the answer below from Dick Sites, the main author of CLD2.

My guess:

There is a concept of priors: the probability that text generally is, say, Danish. As we know, the real datasets and real queries are very imbalanced, they contain much more English than other languages.

Concrete example:

50% of the actual text is English, 1% is Danish.
3% of the actual Danish text is incorrectly detected as English.
3% of the actual English text is incorrectly detected as Danish.
This scenario is very annoying for the hundred of millions of English speaking users of Chrome who will never actually visit a Danish page.

To increase total system accuracy on real world data, you decide to downweight Danish based on the prior probability of Danish. Given the very low prior probability of Danish, let's just say you shut it off completely: 100% of text detected as Danish is just assumed to be English.

Now:
1% of text that is actually Danish is detected as English.
3% of 50% of the text that was incorrectly detected as Danish is now correctly detected as English.

So by just killing Danish completely, real world system accuracy was increased 0.5%.
(3% * 50% - 1% == 0.5%)
Some people in Denmark are unhappy now, but your manager is happy.
You repeat for more languages.

There are difficult tradeoffs. In practice the decision is made to bake the priors into the model one way or another - obviously not down to 0%, just to 1%. This improves overall performance, but has some unexpected effects if you look at small datasets where the imbalances are different than in the global dataset.

Also CLD is made for webpages, not sentences.

Answer 2

CLD2 uses sequences of four letters (quadgrams, or Ngrams with N=4) for most Unicode scripts. The training text came from millions of web pages, filtered by several years' worth of off-and-on software development to use previous versions and high thresholds to remove suspect text, then cross-checked against extensive test data. Plus a lot of manual reading over some weekends of 20,000 lines of output text and its identification. The resulting lookup table has on the order of 1M four-letter combinations (over many scripts: Latin, Cyrillic, Arabic, Hebrew, etc.) each with the top three most likely languages using that combination.

The design center is to detect languages on real web pages. As such, I found that about 10% of all web pages have mixed-language text, but mixtures of only 2-3 languages per page, not dozens. So CLD2 returns the top three languages, with several caveats. A long French sentence containing a single English word is treated as all French. Doing otherwise ends up with a lot of meaningless claims of tiny amounts of random languages. Consider "die" used in an English sentence but claimed to be a tiny amount of German. The threshold for detecting a switch between languages is about 2 sentences, which matches real use on real mixed-language web sites.

So in your example above, give a sentence or two of English, then Indonesian, then English and you should see some combination of English and Indonesian reported; your fake input is just a bit too short.

There is no built-in bias toward English or other common languages as priors, except for a very tiny bias toward the most common language in each script to handle the case of almost no input text. Any available HTML lang= tags or HTTP declarations are used as hints, but they are just hints, overridden by the real text when it differs (tags are wrong about 5% of the time and missing about 50% of the time).

References: CLD2 wiki several articles at
https://github.com/CLD2Owners/cld2/wiki

especially
Why is CLD2 Fast and Small? -- overview
Language Charts -- which languages are statistically close to each other
Language on the Web -- charts of top web sites, showing mixed-language pages among many other things