Ternary diagrams in scatter plot matrix (pairs) with R "compositions" In the documentation for the R compositions package, and in reference to ternary diagrams, it is stated that:

However the ternary diagram can only display compositions of three
  parts. In case of more parts a scatter plot matrix like matrix of
  ternary diagrams is displayed which selects two components against
  some sort of margin of the rest: 

plot(acomp(sa.lognormals5))
plot(acomp(sa.lognormals5), margin = "rcomp")
plot(acomp(sa.lognormals5), margin = "Cu")

In here the author presents this (tantalizingly beautiful) plot:

... without the code!
The "mystery" asterisk $(*)$ is clarified in this passage in Analyzing Compositional Data with R By K. Gerald van den Boogaart, Raimon Tolosana-Delgado

margin = "acomp" (or nothing, the default) computes the third part as
  the geometric of all components except those two from row and colum
  (symbolized with "*").


The question is:

What are the meaning and mathematics behind these deformed circles (lines or curves) generated by the function ellipses, and how to generate them?

 A: The color-coding in the points signify grouping into categorical variable: 
> levels(sa.groups.area)
[1] "Lower"  "Middle" "Upper" 

After much trying and error, I got a practically identical plot with this code:
library(compositions)
data(SimulatedAmounts)
colors = c("gray38", "red", "olivedrab3")

tt = acomp(sa.groups5)

windows(width = 10, height = 10, pointsize = 10)

plot(tt, col = rgb(0,0,0,0), bg = colors[as.numeric(sa.groups.area)], pch = 21, cex = 1.2)

strata = sa.groups5.area
temp = cbind(sa.groups5,strata)

a = acomp(temp[temp[ , 6] == 1, ][,1:5])
ellipses(mean(a), var(a), r = 2, col = colors[1])

b = acomp(temp[temp[ , 6] == 2, ][,1:5])
ellipses(mean(b), var(b), r = 2, col = colors[2])

c = acomp(temp[temp[ , 6] == 3, ][,1:5])
ellipses(mean(c), var(c), r = 2, col = colors[3])



As for the meaning of the curves, or circles around groups of points corresponding to the levels in sa.groups.area...
In Analyzing Compositional Data with R By K. Gerald van den Boogaart, Raimon Tolosana-Delgado the following plot can be found with a telling caption:

and (minimally) paraphrasing:

in which the radius of the lines contain 95% of the probability
  assuming a normal model for the composition and a known variance.

The code of this latter plot likely includes the lines:
r = sqrt(qchisq(p = .095, df = 2))
mm = mean(tt)
vr = var(tt)
ellipses(mean = mm, var = vr, r = r)

... and ?ellipses describes the r parameter in the function ellipses as:
r      a scaling of the half-diameters

