How to prove that NN cannot be fooled with catastrophic outcomes?

Question

This might be a stupid question but I cannot find definitive answer. Is there any way to prove that learned function has some kind of "linearity" with respect to input data and human perception? What I mean by "linearity" here is that small change in input data as detected by human, results in small change in output.

Or vice versa - if we have neural network and an image can we find small change in an image which makes neural network recognise it as completely different image? Something like bad hash function - you change few bytes in a data and get any given hash value.

I think this is important due to security reasons. Say there is a neural network which controls a car. It analyses data coming from cameras and lidars and controls car movement. If we cannot prove NN behave correctly on every possible incoming data sequence then this could be huge security issue. Potentially one can engeneer a situation which looks ok for a human but forces car brains to make mistake and hit wall at full speed.

Same applies to credit decisions - hacker provides crafted data and gets huge credit limit and so on.

Answer 1

Yes, it is possible that small changes, inperceptible for human can cause huge difference in network output, and you can even control the network's output. It is a known issue for neural networks in general. Please take a look at the following papers:

Nguyen A.; Yosinski J.; Clune J. (2015) Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Computer Vision and Pattern Recognition (CVPR '15), IEEE

Goodfellow, I. J.; Shlens, J.; Szegedy, C. (2015) Explaining and harnessing adversarial examples. ICLR 2015