At the near past when I was a student NN concerned as an arbitrary function learner (13 Hilbert problem, Holmogorov stuff like that). But today I am seeing that vanilla Feed Forward NN is no more actual and complicated ones like LSTMs, GRU, CNN etc are come into live. For me this NNs are nothing more than just clever heuristic. But what about NN as arbitrary learners? For example can I take some NN architecture and learn it some data to become, lets say LSTM-like in one experiment and take the same NNs and another data and learn it to become ConvNN?
LSTM, GRU, CNN, NARX etc. are all network architectures. Each of them has certain qualities which can't be learned - shared weights, recurrent connections, self connections, gates - these all must be in place before the learning process starts.
The learning process is aimed at tuning the parameters, but the architecture stays unaffected by this process. Ofcourse, you could combine LSTM and a CNN, but regardless of the input, the network will still act as an LSTM and CNN.
You could have some special training data, that requires LSTM for parts of that data and CNN for other parts of the data - the network will learn to make the outcome of the CNN part of the network less important than the LSTM part (if they are parallel!), but the architecture does not get affected.