I assume you want to compare how good the machine's ordering meets the "real" ordering. In this case, a high rank correlation between truth and machine's output would indicate a successful machine. But how much is enough? The $p$-values to testing of no rank correlation are smaller the smaller the $p$-value is. 0.23 tells you that if the machine's ranking would be completely random (what you know is not true!), the probability of getting results as and more closely (in a probabilistics sense) to the true ordering is 0.23.
As the size of your experiment is each time only $n=5$, you will only get $p<0.05$ iff your machine gets the exact ordering. Even if you swap only two adjacent values, you would only get $p=0.083$. Sadly, only $p<0.05$ would usually be considered as evidence against random ordering of your machine, although this threashold can be questioned. So yes, your approach is not appropriate.
If you want a statistical test, you should rather take the orderings as random samples from the symmetric group $S_5$. Then you can realize at least some sufficient sample size by running your algorithm several times. Unfortunately, I don't know of statistical tests for samples from the symmetric group.
So for the time being, I would use an appropriate mapping of your orderings to a scalar. A metric on $S_5$ or the $p$-values could do the job. Then I would run the algorithm several times with random input and map the found orderings to this scalar. Then I would test if the mean of these sampled scalars is within some appropriate interval around the map of [1 2 3 4 5]. E.g. if you are willing to tolerate adjacent swappings and use the $p$-value as mapping, a mean $p$-value of 0.083 would belong to this interval.