Let's assume you have ordinal data. The problem seems that, with only 1 unit, you can not compute the probability of agreement by chance. There may be a way around this, which has some (non trivial) caveats.
Let's just look at the counts, per scale level, of how many raters agreed on that level. The worse case (for agreement) would be the vetcor $[20,20,20,20,20]$. Overall, the raters disagreed 4000 times. The best case by contrast would $[0,100,0,0,0]$ (or other permutations, where raters all agree on a different level).
So you could replace the "probability of disagreement by chance" by the value 4000 (in your case). Then you can compute e.g. a coefficient (similar to Krippendorff), as $1 - \frac {d_0} {d_c}$, where $d_0$ is the number of observed disagreements, and $d_c$ is 4000. Of course, you could use a similar "worse case" approach with other measures of ordinal interrater agreement.
Caveat; the value (4000) used for $d_c$ is the worse case disagreement. But we know that, in many cases, raters have various biases. E.g. raters could tend to shy away from the extremes (1,5), and instead rate mostly in the "middle" (2,3,4). The value of $d_c$ computed by looking at disagreement accross units takes that into account (i.e. it is not the "worse case", but insted reflects the randomness as observed accross units). So it is not a true Krippendorf alpha (which compares to the natural observed disagreement randomness). But with only 1 unit, we can not observe the natural randomness accross units. So, the next best thing?? So what I am proposing tends to exaggerate agreement. It is up to you to see if that is usable.
