# Can I create a test dataset with known errors to validate accuracy assessment? [closed]

I developed a procedure to measure the geometric accuracy of 3D building models based on the similarity to a 3D point cloud. Therefore I created mainly two quality criteria. The result of my automatic method is an index for each building how well the model fits the point cloud. I am interested in how to validate this accuracy result and if this is necessary.

Is it possible to alter a test dataset that fits the point cloud according to the quality criteria so that it contains errors? I then would want to use this "faulty" test dataset to see whether the errors are correctly detected. I am afraid that this method could be argued as too biased because I am introducing the errors. Error creation would be done either manually by hand or automatically e.g. by a shifting algorithm that changes the models position.

Edit for clarification:

• 3D point cloud means a set of points, each with x-, y-, z-coordinate
• 3D-Model is a mesh model, represented by vertices, which are also points with x- y-, z-coordinate

I am looking on the building from the top view and arrange the outline inside a regular grid. For each grid cell, I am calculating the mean/median of z-coordinates inside. For the points of the building model as well as for the points of the point cloud. After I have this 2D representations in the xy-plane, I am calculating the differences between each cell. These are my "distance" measures. I now apply a threshold to the calculated distances to decide whether they are significant in terms of "too high". Without going more into details, each building gets assigned a percentage value, how many cells contain differences considered as erroneous. In fact I am using some more criteria, but those are not relevant for the question.

What I want to know: If I have a set of 3D building models, that are all correct in terms of my accuracy check (all differences below threshold). Is it valid to change the correct building models so they contain errors in terms of my threshold and use them to test my accuracy check? I want to validate my developed procedure if it is able to detect the erroneous buildings. Otherwise my percentage values are being calculated but it is not clear if the procedure is the right way to do it.

• "fits the point cloud according to the quality criteria so that it contains errors" -- can you clarify this? What constitutes an error? – shadowtalker Oct 28 '18 at 22:27
• Basically accurate fit means low distances between points of the point cloud and surfaces of the model. In general a sufficient fit is defined by distances below a specific threshold. So I would like to introduce errors that lead to increased calculated distances which in turn lead to rejected fit according to my quality criteria. I want to validate if my developed procedure can detect this errors. – conste Oct 28 '18 at 22:35
• Thanks for clarifying. Are you asking if this is generally an acceptable thing to do in general, or are you asking for advice on specific procedures for doing it? – shadowtalker Oct 28 '18 at 22:38
• To be honest I am looking for general advice if this method can be considered serious or if it is a problem that I am creating synthetic errors. But - if there are any procedures applicable to this problem, I would be happy to get to know them. Unfortunately I couldn't think of any. – conste Oct 28 '18 at 22:41

Yes, this is a perfectly valid procedure. You need to make sure that your model can distinguish between "similar" and "not similar". So you simulate both kinds of datasets, and make sure that they are flagged correctly.

You can also take a "similar" dataset and perturb it. This is a form of sensitivity analysis. It's also a good way to test the robustness of a model, by taking a "similar" dataset and perturbing it in different ways. The perturbations can be deterministic or randomized; I recommend both. Here are some general recommendations for sensitivity analysis in machine learning models that might apply to you.

Introducing errors/perturbations by hand isn't a bad thing. It only introduces "bias" in the sense that you will only be able to evaluate robustness against errors you can think of. That's fine; something is better than nothing. However, I don't recommend going back and adjusting your model in response; not all errors are equally probable, and it's possible that the errors you manually introduce aren't representative of the errors your model will encounter in the wild. If possible, see if you can figure out what kinds of errors you are most likely to encounter. Then you can (at least heuristically) know if you need to worry about them, and use that knowledge to choose the errors you introduce.

Your 3-D model, reasons for simulation, and criteria for declaring an error are all very vague, so it is difficult to give a direct answer.

For specificity, suppose the constructed objects are intended to be approximately cubical, and that the width, height, and depth are independently $$\mathsf{Norm}(\mu=10,\, \sigma=2).$$ Then the volumes of the cubes should be about be about 1000 cu. units. Perhaps an object that has less than 600 cu. units is unacceptable.

Provided your model for the volumes of the containers is correct, by simulating the volumes v of 100,000 cubes, you could get a good idea what proportion of the units will be unacceptable. A brief simulation in R shows that about 11% of the containers will be unacceptable. So, if you're seeing 30% unacceptable units, either something is wrong with your model or with construction methods. [In this case, the geometry is straightforward and an analytic answer using calculus would not be difficult. For more complex models, an analytic solution might not be so easy, and simulation might be an attractive method.]

set.seed(1028);  m = 100000
w = rnorm(m, 10, 2);  h = rnorm(m, 10, 2);  d= rnorm(m, 10, 2)
v = w*h*d;  mean(v);  mean(v < 600)
[1] 998.7257  # average volume aprx 1000
[1] 0.11365   # proportion unacceptable


Here is a histogram of the volumes of containers made according to the specified model.

If the 11% percentage of unacceptable containers is too high, you might investigate whether it is feasible to decrease the standard deviations of the three dimensions to 1.5 or 1.0, instead of 2.0. And then simulate that model.

If you had something very different in mind in your Question, please try to be more specific about what you have in mind. What feature of the simulation above is not on target? Then maybe someone can give a more useful answer.

• I am not sure if this is a comparable setup. I am calculating distances between a 3D model and the points of a point cloud. In fact I am splitting up the model in a grid structure in the xy plane and compare mean heights between the models points and point clouds points inside each cell. I am using a threshold if the difference inside a cell is considered significant. Based on that I am applying an percentage value to the building how many cells are considered as an error. Now I want to know if it is valid to create a test data set where some models would be considered falsy to check my test. – conste Oct 28 '18 at 23:40
• Sorry, still can't understand your procedure or goals. What is 'point cloud'? 'Mean heights' of what? What measure of 'distances between'? What are criteria for 'significance' and 'error'? What is the point of simulating a model you know to be wrong? Suggest reorganizing original Question, so others can know what you're doing. – BruceET Oct 29 '18 at 7:53
• Updated my question. – conste Oct 29 '18 at 10:26