The statistical issue might be easiest to think about in terms of finding the true frequency of heads for a coin that might be biased (binomial distribution). If you only flip a fair coin 3 times per experiment, then in one out of eight experiments you will get all heads, and in one out of eight experiments you will get all tails. That is, in one-quarter of the experiments with a fair coin you will find an empirical frequency that is as far away as possible from the true frequency of 1/2 for heads, if you only sample 3 times in your experiment.
You might think analogously about the case of 16 distinct possibilities (a multinomial distribution) sampled only 40 times. With not even 3 times as many samples as there are possible outcomes, you will not have much assurance that the observed frequencies for each possibility are close to the true underlying frequencies of the population. For example, S. K. Thompson examined sampling from multinomial distributions (The American Statistician, Vol. 41, No. 1, 1987, pp. 42-46; available through JSTOR if you have that access). Based on his table, with only 40 samples, half the time in a worst-case scenario at least one of the observed probabilities $p_i$ will be more than 0.1 unit away from the true probability. (If each of the 16 possibilities is equally likely, each true probability would be 0.0625.)
In principle, to "draw a line" at $N=40$ might have been based on a judgement by the authors about the error that such limited sampling would make in their calculation of "epipolymorphism." In this case, however, the requirement for $N>40$ might instead represent an attempt to omit technically questionable results rather than to achieve some strictly statistical purpose.
Data analyzed for "epipolymorphism" in the paper were reduced representation bisulfite sequencing (RBBS) results. This method determines whether certain cytosine residues next to guanines (CpG) in DNA have been methylated, one type of "epigenetic" modification of different parts of the genome. As next-generation sequencing (NGS) was used for RBSS, several consecutive CpG sites could be analyzed on each single piece of DNA. The authors examined thousands of sets of 4 consecutive CpGs (4-mers) across the human genome and determined how the variability of methylation patterns for the 4-mers changed as a function of their average CpG methylation, among many biological samples.
If all 4 CpGs in a 4-mer are completely methylated or completely unmethylated then there is no variability (0 epipolymorphism) but at intermediate average methylation there are distinct, biologically interesting, possibilities. For example, 25% average CpG methylation of a 4-mer could come from one out of every 4 DNA pieces having all 4 of its CpGs methylated, or all DNA pieces having one identically placed CpG methylated out of the 4, or something more random providing 25% average methylation. The "epipolymorphism" (noted in a comment by @Sycorax to be the same as Gini impurity) was evidently intended to capture the variability in this epigenetic modification among individual pieces of DNA covering each analyzed 4-mer in the genome, a type of epigenetic heterogeneity among cells.
The supplementary data for the paper suggest that 300 to 600 individual pieces of DNA ("reads") were typically analyzed for each 4-mer for each biological sample. Having no more than 40 reads available for a particular 4-mer suggests substantial under-representation of that 4-mer (only about 1/10 of the average representation) in that DNA sample. RBBS involves several processing steps including the polymerase chain reaction (PCR), which can have different amplification ability on different regions of the genome.
Omitting under-represented genomic regions in NGS can be a way to try to avoid regions of the genome that might have been affected by technical limitations. The hard cutoff at $N>40$ might have been Procrustean but it was a simple way to proceed; it still left tens of thousands of 4-mers to examine in almost all samples. Clearly the 4-mers with low representation would also have been less reliable in terms of identifying their methylation patterns, but I would guess that the authors simply looked at the distribution of 4-mer coverage among all genomic sites and found that those with $N\le40$ appeared to be technical outliers.
The authors of the paper could presumably clarify how they came up with that cutoff.