Neural networks assume continuity, what does this mean?

Question

I encountered the following paragraph by Pedro Domingos (mentioned in Gary F. Marcus paper):

ANNs assume continuity, graphical models assume conditional independence, and instance-based learning assumes similarity; and correspondingly, neural nets make it easy to incorporate types of continuity like translation invariance, graphical models [make it easy to incorporate] conditional independences, and [instance-based models make it easy to incorporate] knowledge of what makes things similar (in the kernel or distance measure, which will vary with the domain).

I understand conceptually what is translation invariance, but I don't understand what is meant by continuity and why translation invariance is a kind of continuity.

Answer 1

A continuous function is (very informally), one you can draw/plot on a graph without lifting your pen. See this wikipedia page for the gory details.

If you implement a translation function: $x' = \text{translate}(x, \Delta)$ which takes image $x$ and translation amount $\Delta$, using bilinear interpolation, then it will be continuous wrt both $x$ and $\Delta$. Let $f(\cdot)$ be a continuous neural network. Then $g(x,\Delta) = f(\text{translate}(x, \Delta))$ is continuous wrt both $x$ and $\Delta$ since continuity is closed under composition.

This is all to say that since neural networks are continuous with respect to the input image, they will also be continuous with respect to the amount of translation, which of course is a prerequisite for being translation invariant (keep in mind neural networks aren't really translation invariant in the formal sense).

Anyway I think the connection between continuity and translation invariance is quite shaky in the paragraph you quoted... I wouldn't read into it too much.