To my understading what a 1x1 convolution does is gives an embedding of the (i,j)th entry of the feature map along its depth. Besides here some dimensionality reduction is also done. How will the scenario change when we have 5x5 or 3x3 convolution done followed by a 1x1 convolution. I know that the number of computations will blow up. But is their some other point that I am missing?
You are right.
The whole point of using 1x1 convolutions is to reduce the dimension along the direction of the number of channels while - keeping other dimensions same, not losing lots of useful information and not having to learn lots of new parameters to do this.
And if we still want to do 5x5 or 3x3 convolution directly, besides the issue of computational cost, semantically, when number of filters gets unreasonably large, for many of the filters patterns being learned can be either are repetitive or are not of much use for our data and problem at hand (Assuming we are looking at the right place in the network). And doing 1x1 convolution after this would be somewhat similar to doing the whole bunch of not much useful computation first and then getting rid of all that computation.
Suppose your previous layer outputs $h\times w \times d, 1\times 1$ would take it to $h\times w\times k$ where ideally $k<d$. The picture you are showing is of the inception module where the basic assumption is that features exist at many different scales. A filter of $3\times 3$ cannot capture a feature in a $5\times 5$ window. And a $5\times 5$ filter has a hard time modeling a $3\times 3$ filter. So we try to combine patterns on different scales. Because we are doing so much computation, and then joining all this information, we dont want to be very large hence we do a $1\times 1$ before so that we reduce the dimension.
