To answer from a different perspective, one philosophical point may help: the concept of learning in deep learning.
In deep learning, there are multiple learning steps. In the first step, the data input is 'converted' (or learned) into a synthetic intermediate output (a bit higher abstraction, loosely speaking). Then in each step, the previous output is progressively learned (or 'transformed') into higher abstraction features, which may or may not be comprehensible to humans. Loosely speaking, the combination of these layers will approximate the 'model' for you, we don't need to specify any model or hypothesis beforehand. This is the same 'learning' concept as in cognitive science: find/construct higher abstraction from raw input.
So, do we need NN for deep learning? Yes, in practice. With my limited knowledge, I would say this is the most convenient and efficient way to do it. In theory, it depends. If you build a 'learning' framework in which your model can create 'deep' abstraction (iteratively increasing) from raw input to achieve the task at hand, it may count as deep learning.
In practice, it's a bit simplistic to view Deep Learning as just a vanilla multi-layer neural network. More than often, the workflow includes different NN-components and other transformation layers, each part can be wildly different from the next.
I hope this answer provide some useful insights without the need to use technical terms.