I once studied this with a classroom. Students were learning how to use a mouse to digitize features shown in an aerial photograph. The features consisted of building and waterbody outlines. At my request, some students submitted a file of the waterbody (giving sequences of coordinates as they had indicated them). I compared the files to one another to study their variability.
The figure overlays seven attempts by six students to digitize a grayscale image of a lake.
I have learned--from this experience and from extensive experience in digitizing images and graphics like the one in your experiment--that the accuracy depends on many factors. They include how well the target object can be discriminated (which is a function of graphical attributes such as line thickness, intensity, saturation, color, pattern, and aliasing), the "feel" and precision of the mouse or digitizing puck, the scale of the object as shown on the screen, the resolution of the screen, and even geometric variations in the target feature. Notably, people tend to pay more attention to, and give more accurate renderings of, near point-like parts of a shape, such as a corner, cusp, or endpoint (of a linear feature). This distinguishes the square (with four corners) from the circle (with none).
There are many ways to approach the measurement of digitizing error. In this situation, we might attempt to capture what the participants actually see. They are likely comparing the computer's rendering of the polyline (as they have constructed it so far) on top of its rendering of the target feature. Because they are seeing the difference between two raster images, we ought to measure any discrepancies in terms of those images rather than in terms of the vector (sequence of coordinates) representation of the digitization process. Those discrepancies will, at the end, be visible as gaps between what was digitized and how the target was represented on the screen. In this experiment, since closed linear features were digitized, it is therefore tempting to construct a polygon bounded by the target feature and the digitized feature, represented in raster format exactly as seen on the screen. Various properties of this polygon (which will be a skinny, tortuous, disconnected set of pixels along and surrounding the target) will serve to measure overall error. The simplest metric will be its area, but other useful ones can be obtained by analyzing the polygon's medial axis transform, or "skeleton." With this you can assess the maximal deviations and develop measures related to how much the hand might have been shaking during the digitization.
This figure compares two digitized traces (red and cyan arcs) in terms of the image pixels lying between them, shown in gray. The sum of the gray pixels (their total area, essentially) is a simple, reasonable measurement of the discrepancy between the traces.
Notice that where the traces would be seen to coincide in the original image, there are no pixels between them. Thus, perfect coincidence as seen by the subject would result in an empty polygon and a zero sum, even though it is not mathematically possible for any finitely digitized polyline to make a perfect circle (or any other curve).