Neural networks assume continuity, what does this mean? I encountered the following paragraph by Pedro Domingos (mentioned in Gary F. Marcus paper):

ANNs assume continuity, graphical models assume conditional
  independence, and instance-based learning assumes similarity; and
  correspondingly, neural nets make it easy to incorporate types of
  continuity like translation invariance, graphical models [make it easy
  to incorporate] conditional independences, and [instance-based models
  make it easy to incorporate] knowledge of what makes things similar
  (in the kernel or distance measure, which will vary with the domain).

I understand conceptually what is translation invariance, but I don't understand what is meant by continuity and why translation invariance is a kind of continuity.
 A: A continuous function is (very informally), one you can draw/plot on a graph without lifting your pen. See this wikipedia page for the gory details.
If you implement a translation function: $x' = \text{translate}(x, \Delta)$ which takes image $x$ and translation amount $\Delta$, using bilinear interpolation, then it will be continuous wrt both $x$ and $\Delta$. Let $f(\cdot)$ be a continuous neural network. Then $g(x,\Delta) = f(\text{translate}(x, \Delta))$ is continuous wrt both $x$ and $\Delta$ since continuity is closed under composition. 
This is all to say that since neural networks are continuous with respect to the input image, they will also be continuous with respect to the amount of translation, which of course is a prerequisite for being translation invariant (keep in mind neural networks aren't really translation invariant in the formal sense).
Anyway I think the connection between continuity and translation invariance is quite shaky in the paragraph you quoted... I wouldn't read into it too much.
