# What is this distribution? Inverted S / cursive N

I have come across a graph pattern in two basically unrelated experiments, and I want to understand where it is coming from, or at least how to handle it statistically.

I work in computational linguistics; I have picked up some statistics along the way, but barely enough to make grunt noises.

For evaluating a language-identification system, I ran an experiment on a subset of the Leipzig corpus collection. I got the following graph (I will describe it in more detail below):

(The red "similarity" plot is the beef here; the other plots are secondary measures.)

I ran the same experiment on the 20 Newsgroups corpus. Here is a graph from that:

While the sigmoid shape is less pronounced here, it appears (from a collection of graphs not shown here) to be yet another in a series of similar shapes.

More recently, I was looking at the entropy of articles in the Reuters v2 multilingual corpus and vaguely expected something like a Zipf distribution perhaps. But lo, I ended up with a fairly similar graph again:

I notice that all of these exhibit an inverted S or cursive N shape. I googled a bit for "inverted S" and that suggested me that what I am looking at is a logistic distribution, albeit with the axes inverted. Do you agree, do you think it would be useful for me to model this as a logistic distribution somehow?

Most of the introductory materials I could find only cover the use of logistic distributions for yes/no or multiple choice problems, which these are emphatically not. However, I stumbled across http://www.wmueller.com/precalculus/families/1_80.html which I thought was useful (also the previous and following pages) and which teaches me that "logistic functions model resource limited exponential growth" (and also relate them to "accumulation of a function with a peak value").

Ideally, I would like to obtain this sort of level of understanding of the phenomenon I am looking at. What causes these graphs to take this form? Then what would be a good way to model the phenomena? Is there actually some sort of resource constraint at work (I am vaguely thinking of linguistic models such as optimality theory -- that the constraint could be one of economy of expression given the available communication bandwidth, perhaps balanced against the need for expressiveness and redundancy, or some such)?

## Dendrogram graphs

The first two graphs basically depict a dendrogram. I used a language identification tool (TextCat) to calculate the pairwise similarities of my items (sentences in the Leipzig corpus, articles in the 20 Newsgroups corpus) and ran an agglomerative clustering algorithm to produce a dendrogram. The X axis is the index into the dendrogram; at the extreme left, we have a single cluster and a threshold similarity of 0%. The first point on the X axis corresponds to the least similar item in the dendrogram; if we split at this point, we get two clusters (or rather, a singleton outlier and all the other items in a big whopping cluster) and the similarity on the left Y axis. The further we proceed, the more clusters we will have, and the higher the similarity threshold below which we will keep two items in the same cluster.

The graphs include the average similarity, the standard deviations from the average similarity, and the number of clusters of various sizes at each point, against the right-hand Y axis. (We regard singleton outliers as "clusters" in this regard.)

(I'm afraid it's been too long since I did this that I could explain the secondary graphs even to my own complete satisfaction. As we proceed towards the right, the number of clusters with more than one member should start decreasing; at the extreme right, every item should be in its own cluster. Maybe I just made a mistake when I created that plot.)

When I started this project, I expected to find a convenient elbow somewhere along the similiarity plot which would suggest a suitable number of clusters for the samples. I expected the clusters to be something like subtopics or some such. However, when I saw the graphs, I focused instead on the extreme ends of the chart -- the outliers near the left end, and the very similar samples towards the right end. This proved fruitful for weeding out misclassified samples (!) on the left, and removing near-duplicates near the right, but in the end, I could not find an obvious way to decide how to cluster my samples. I ended up using multiples of the standard deviation to cut off the extremes.

Now, I would hope to find a better model than the linearity of the standard deviation, but I realize that a logistic (or similar) fit actually means that at least part of the items near the extreme ends may well be legitimate, so this kinds of complicates matters for me. Not all the samples at the extreme left were misclassified, so this is perhaps a more satisfying model in any event.

## Entropy graph

The third graph is much simpler. It shows the byte-level entropy in each individual language in a corpus of newswire articles. I simply calculated the entropy for each individual article, sorted by that measure, and plotted the result. The X axis is normalized to a 0:1 range; for example, for the Dutch graph, there are 1794 data points, whereas the Russian graph represents 17486 samples.

Some of the jaggies I think can be explained by aberrations in the material. In particular, I think the sudden jump in the Russian entropy towards the right end is caused by the fact that a large number of articles in this collection are in fact in English, not in Russian, so the far end describes the entropy of English instead of Russian.

Similarly, but on less solid evidence, I speculate that the jaggies on the left of the Danish graph are because of quality problems (near-duplicates, corrupted articles, etc).

• Sorry for the crude graphics; I have yet to graduate from Gnuplot (shudder). Dec 4 '13 at 0:04
• The shape is characteristic of the arcsin cumulative distribution function for random walks.en.wikipedia.org/wiki/Arcsine_distribution
– pat
Dec 4 '13 at 0:49
• Or a beta distribution when $\alpha=\beta$ and $\alpha<1, \beta<1$.
– Sycorax
Dec 4 '13 at 1:05

I finally managed to figure out this "inverse sigmoid" graph. It's quite simple, really. My charts show the actual raw data, whereas I imagine a "real statistician" would immediately go look at the distribution, which is a more useful graph anyway, really.

Here are a couple of new graphs from another part of the Leipzig corpus collection (namely, the Turkish 2011 newscrawl 10K corpus). The first one is the sample length, the second, the entropy for each sample.

(In each case, I sorted the samples by the measure I was plotting, so a sample's index on the x axis in the first chart is bound to be different in the second chart in the vast majority of cases.)

Here are the corresponding distribution histograms.

So in other words, the slope of the inverse sigmoid indicates the heights of the corresponding histogram bars. A steep slope means the adjacent samples are not very similar, so you get just a few samples in each column in the histogram. When many adjacent samples are similar, indicated by a flatter slope, they end up in the same column in the histogram, so that's a higher bar.

The distribution diagram immediately tells you about the shape and skew of the distribution, whereas the inverse sigmoid only vaguely hints at a more or less standard (let alone Gaussian normal) distribution -- you can't really tell from the graph in any more detail.

The reason I plotted my data this way in the first place was that I wanted to be able to map back from a point on the chart to an individual sample I could look at more closely; but I see now that this isn't really "statistics" at all; and also, that actual statistics make sense to use in this analysis.