Why would quartile plotted against percentile not be a straight line? In the famous Dunning-Kruger paper (ungated) perceived and actual scores are plotted, with quartile on the x axis and percentile on the y axis.  The message is the perceived scores, and the actual scores are shown as a line for reference.  In figures 1 and 4 the actual score line is straight, as one would expect as we have the rank score for the same value on both axis. However figure 3 has the 2nd quartile point significantly lower than the identity line, and in figure 2 the 2nd and 3rd quartiles are slightly lower.
What is this describing?

 A: I agree this is odd, but I suppose it could be caused by tied scores showing up in particular quartiles. Suppose we have 100 subjects that get scores from 1 to 100 in 1-point intervals, except that everybody who would have had a score between 26 and 50 instead gets a score of 30. In this case, the average score percentile among the second quartile is either 26 or 50, depending on how you handle endpoints in the percentile calculation. For a dataset with only distinct scores, the average percentile of the second quartile must be 37.5, but it would be possible to find a different average percentile within a quartile that contains tied scores. Not sure if this is the case, or if it's just an error in plotting - even Figures 1 and 4 don't look exactly the same in "actual test score" values (the second quantile in Fig 1 is near 35, but it's closer to 40 in Fig 4).
A: You said: "The message is the perceived scores, and the actual scores are shown as a line for reference" but I disagree.
It doesn't matter that they aren't straight.
Consider this, a variant of the upper left subplot:

The yellow-colored region is where the perceived ability is higher than the actual ability.  The green region is where the perceived ability is lower than the actual.
I think the important point is that when actual skill is low, estimate of performance is falsely very high, but this decreases as skill increases until the skill is very high, and estimated skill is moderately lower than actual skill.
Distribution of the measured variable can vary with population.  It is easy to make a pathological example, where 25% of values get grades of 10% and the remainder get 100% to make a curve that is concave down.  Similarly if 75% of all grades are 10% and the last quarter are 100% then it is strongly concave up.  There are many general forms of curves and only a subset are linear.
Here is some code for exploring this a little bit, numerically.
#library stuff
if(!require(pacman)) {
install.packages("pacman"); 
require(pacman) }

p_load(dplyr,     #munging
       ggplot2)

#reproducibility
set.seed(575743)

#uniformly random sample of grades
y <- runif(100, min=0, max=100)

# y <- rnorm(100, mean=80, sd=10)
# y <- ifelse(y>100,100,y)

#make background points
q_list_all <- numeric(100)
q_store <- data.frame(q=0:100, y=0*(0:100), bin=0)
for(i in 1:101){
  q_val <- (i-1)/100
  q_store$y[i] <- quantile(y,q_val)
}

q_store$bin <- ceiling(q_store$y/round(diff(range(q_store$y))/3.5))

#make the plot
ggplot(data=q_store, aes(x=q, y=y) ) +
  geom_path(color="blue") + geom_point(color="blue") +
  geom_boxplot(aes(group=bin), alpha=0.5) +
  xlab("Quartile") + ylab("grade on test") 

This is the plot that it makes:

If instead of using a uniform distribution, you were to use a normal one centered at 80% with a standard deviation of 10% and a maximum of 100%, then the plot looks more like this:

