Why do we connect convolution layers in sequence instead of applying them separately on input image? I am already aware of the convolution function, CNN and all. I have already implemented a few. But this question strucks my mind every time. Most of the networks I have seen, use a stack of convolution layers. That is, the current convolution layer works on the output of previous convolution layer and so on.
So, is there a specific reason for doing this? Alternatively, we can also apply each convolution layer of different kernel size individually on the input image and then use their outputs in some way to get what we want. Why not do this?
Like is there a reason behind this? Or is it like just following people blindly?
 A: The reason for stacking convolutional layers is that usually complex features consist of less complex features: e.g. a cat has a face, the face has ears, the ears have edges.
Loosely speaking, convolutional layers close to the input layer detect less complex features and convolutional layers closer to the output layer detect more complex features, by combining the simpler features from the previous layers.
You are right in thinking that applying different kernel sizes "in parallel" is a good idea but you are wrong in assuming that researches are "following people blindly".
2014 the GoogleLenet/Inception architecture was introduced and won the ImageNet competition. It uses parallel convolutions with different kernel sizes. The idea behind that is that the network gains flexibility in processing "sparse" or "dense" information. (Imagine you want to classify dog breeds, and some images are zoomed in on the dog, while in other images the dog occupies only a small fraction of the image. This would be a use case for parallel convolutions)
