# How to calculate confidence in AI algorithm

I noticed that the eye-tracking data that I am working with had a bias that caused fixations (where people are looking) to be translated away (north in this case) from the true location of Points-of-Interest on the screen (see the left panel). In my image, the filled circles are all Points-of-Interest. The outlined ovals are where the person was supposedly looking (fixations). The lines aren't important for this problem. Before and after position adjustment.

I wrote a quick AI algorithm with a minimum-distance heuristic to correct for this bias. Applying the same transformation to all 180 of my data points brings all the fixations in line with a point-of-interest.

I'm looking for a confidence interval that says that the chances that this transformation aligns fixations with point-of-interest by chance is very low, and that this is likely where they were actually looking.

Is there a test that would give me a good confidence metric for this situation?

• For those of us who aren't familiar with eye-tracking research, it would help if you could explain how to interpret the images. Then, what exactly is the "this" that's supposed to be not a coincidence? Jul 21, 2017 at 0:57
• Sorry, the filled circles are all objects on the screen (Points-of-Interest). The outlined ovals are where the person was supposedly looking (fixations). The lines aren't important for this problem. As you can see in this example, the data is slightly north of where things are on the screen. Applying the same transformation to all 180 of my data points brings all the fixations in line with a point-of-interest. This can't be a coincidence. I'm looking for a confidence interval that says that this transformation is meaningful. Jul 21, 2017 at 2:14
• Okay, that helps. Now, are you thinking of this as a data-cleaning step ("we found indications that the eye-tracker was miscalibrated and corrected that by applying this transformation") or an analytical step ("this transformation brings all the fixations in line with a point-of-interest, which indicates that ...")? Jul 21, 2017 at 12:54
• Data cleaning step. Jul 21, 2017 at 14:04
• I've made some edits to the title and tags that should do a better job of getting attention from the right folks. Jul 21, 2017 at 14:34

The kind of justification you want to give depends on the way the transformation works and the overall goals of your study. For example, if you're building an eye-tracking device, and writing up a paper showing how accurate your device is, you don't want to apply any transformation! On the other hand, if you have a simple transformation (e.g., $(x,y) \mapsto (x+c_x, y+c_y)$ for constants $c_x$ and $c_y$) and this doesn't change anything substantial but makes your data analysis easier (e.g., reducing ambiguity about what fixation is associated with what point-of-interest, which makes it easier to interpret the final results), then you may need little or no justification.