Regarding model capacity
I'm going to go with the first reason you state and explain it a bit better. There should a sort of balance between model capacity and dataset size. If the capacity of a model is much higher than necessary$^1$ for a particular dataset, then that model will overfit to it.
The extreme case is when a model's capacity is so high that it can completely memorize the training data. This means that the model has learned the data so well that it achieves zero error. Obviously, for a given network, the smaller the dataset, the easier it is for a network to memorize.
Now, because neural networks typically have a large number of parameters (i.e. high capacity), it is very easy for them to memorize small datasets. Furthermore, even if it can't completely memorize the data, it will certainly overfit on it.
On the other hand, if we have a large number of samples, it will be very hard for the model to memorize them. In order to improve its performance, the network will actually have to identify the relevant information and model it.
Thus the problem isn't as you state that the network can't learn from small datasets, but that it's too easy to learn from them.
$^1$ This is pretty arbitrary. The necessary capacity is one that allows the model to learn all useful relationships between the data and their labels but not the underlying noise in the data.
Another way to view this is through the lens of dimensionality. Say you have a very simple problem to solve (e.g. the XOR problem). You can actually need $4$ samples to solve this. Why? because it's a very low-dimensional and easy task.
However as the dimensionality of the data increases you need a much higher number of samples to sufficiently represent the feature space.
Given the same number of samples, as the dimensionality of the data increases, they become exponentially sparser. This causes the data to be easily separable by a classifier (i.e. in the figure above it's much higher to distinguish between green and red dots in the ligh-dimensional space). This is up to a certain point a good thing because its easier to learn. However, it's also much easier to overfit on high-dimensional datasets!
For example, say you want to classify $256\times256$ RGB images. This data has a dimensionality of $256 \cdot 256 \cdot 3 = 196608$. To properly learn on this data you need a large number of samples, or else the data won't sufficiently represent the vast feature space and high-capacity models can overfit with ease on this dataset.
If you have few samples distributed in a large feature space, then it's easy for a model to memorize them (or at the least overfit). The higher the capacity of a model, the easier it is for it to overfit.
So if you have a high dimensionality and a high-capacity model you need a lot of data.
When training deep neural networks we have to control the size of the gradients during backpropagation. There are two prominent issues here which I won't go into in detail: vanishing gradients and exploding gradiets.
The main idea is that you want the magnitude of the gradients to remain constant as they are backpropagating through the net. A popular way to combat this is by initializing the weights of the network in smarter ways. Two initialization techniques have proven to be the most popular: Glorot initialization (sometimes referred to as Xavier initialization) and He initialization, both of which are variations of the same idea.
In order to work properly both assume that the input data is standardized. So the magnitude of the weights of your model are initialized under the assumption that your data is centered on zero and have a unit variance. It's not a good practice to feed data that is a couple of orders of magnitude larger than that.