Can I create a test dataset with known errors to validate accuracy assessment? [closed]

Question

I developed a procedure to measure the geometric accuracy of 3D building models based on the similarity to a 3D point cloud. Therefore I created mainly two quality criteria. The result of my automatic method is an index for each building how well the model fits the point cloud. I am interested in how to validate this accuracy result and if this is necessary.

Is it possible to alter a test dataset that fits the point cloud according to the quality criteria so that it contains errors? I then would want to use this "faulty" test dataset to see whether the errors are correctly detected. I am afraid that this method could be argued as too biased because I am introducing the errors. Error creation would be done either manually by hand or automatically e.g. by a shifting algorithm that changes the models position.

Edit for clarification:

• 3D point cloud means a set of points, each with x-, y-, z-coordinate
• 3D-Model is a mesh model, represented by vertices, which are also points with x- y-, z-coordinate

I am looking on the building from the top view and arrange the outline inside a regular grid. For each grid cell, I am calculating the mean/median of z-coordinates inside. For the points of the building model as well as for the points of the point cloud. After I have this 2D representations in the xy-plane, I am calculating the differences between each cell. These are my "distance" measures. I now apply a threshold to the calculated distances to decide whether they are significant in terms of "too high". Without going more into details, each building gets assigned a percentage value, how many cells contain differences considered as erroneous. In fact I am using some more criteria, but those are not relevant for the question.

What I want to know: If I have a set of 3D building models, that are all correct in terms of my accuracy check (all differences below threshold). Is it valid to change the correct building models so they contain errors in terms of my threshold and use them to test my accuracy check? I want to validate my developed procedure if it is able to detect the erroneous buildings. Otherwise my percentage values are being calculated but it is not clear if the procedure is the right way to do it.

Answer 1

Yes, this is a perfectly valid procedure. You need to make sure that your model can distinguish between "similar" and "not similar". So you simulate both kinds of datasets, and make sure that they are flagged correctly.

You can also take a "similar" dataset and perturb it. This is a form of sensitivity analysis. It's also a good way to test the robustness of a model, by taking a "similar" dataset and perturbing it in different ways. The perturbations can be deterministic or randomized; I recommend both. Here are some general recommendations for sensitivity analysis in machine learning models that might apply to you.

Introducing errors/perturbations by hand isn't a bad thing. It only introduces "bias" in the sense that you will only be able to evaluate robustness against errors you can think of. That's fine; something is better than nothing. However, I don't recommend going back and adjusting your model in response; not all errors are equally probable, and it's possible that the errors you manually introduce aren't representative of the errors your model will encounter in the wild. If possible, see if you can figure out what kinds of errors you are most likely to encounter. Then you can (at least heuristically) know if you need to worry about them, and use that knowledge to choose the errors you introduce.

Answer 2

Your 3-D model, reasons for simulation, and criteria for declaring an error are all very vague, so it is difficult to give a direct answer.

For specificity, suppose the constructed objects are intended to be approximately cubical, and that the width, height, and depth are independently $\mathsf{Norm}(\mu=10,\, \sigma=2).$ Then the volumes of the cubes should be about be about 1000 cu. units. Perhaps an object that has less than 600 cu. units is unacceptable.

Provided your model for the volumes of the containers is correct, by simulating the volumes v of 100,000 cubes, you could get a good idea what proportion of the units will be unacceptable. A brief simulation in R shows that about 11% of the containers will be unacceptable. So, if you're seeing 30% unacceptable units, either something is wrong with your model or with construction methods. [In this case, the geometry is straightforward and an analytic answer using calculus would not be difficult. For more complex models, an analytic solution might not be so easy, and simulation might be an attractive method.]

set.seed(1028);  m = 100000
w = rnorm(m, 10, 2);  h = rnorm(m, 10, 2);  d= rnorm(m, 10, 2)
v = w*h*d;  mean(v);  mean(v < 600) 
[1] 998.7257  # average volume aprx 1000
[1] 0.11365   # proportion unacceptable

Here is a histogram of the volumes of containers made according to the specified model.

If the 11% percentage of unacceptable containers is too high, you might investigate whether it is feasible to decrease the standard deviations of the three dimensions to 1.5 or 1.0, instead of 2.0. And then simulate that model.

If you had something very different in mind in your Question, please try to be more specific about what you have in mind. What feature of the simulation above is not on target? Then maybe someone can give a more useful answer.