Besides convnet and stacked RBMs, are there other noteworthy deep NN models?

Question

It seems to me that ConvNet is only useful when applied to convolutional data like images and sound, where data has a continuous nature (data points that are near often have similar values or correlated).

The other deep network model I know is Hinton's stacked restricted Boltzmann machines, where each layer is like an autoencoder and you pre-train it layer by layer.

And in ConvNet the current state of art is that you don't pretrain, just use ReLU neurons and brute-force backprop will do. So the only innovation in training algorithm is the stacked RBMs.

After these two successes, are there any other noteworthy models that has developed new training algorithms or new structures? (I know there are models like neural turing machines and long-short term memory but they look very artificial and there's no reason to expect they can do well, unlike ConvNet where convolution and pooling can be somehow explained so it feels natural.)

For example when the data is NOT continuous, for example you have many indicators of a certain stock value, are there good deep NN models to use in this case?

Answer 1

Convolutional Neural Networks can also be applied to text, e.g. A Convolutional Neural Network for Modelling Sentences, see also Understanding Convolutional Neural Networks for NLP.

RBM pretraining is essentially obsolete (though, maybe not completely). In particular, layer-wise pretraining was superseded by new techniques.

long-short term memory but they look very artificial and there's no reason to expect they can do well

LSTM was designed with concrete motivation of handling vanishing gradients problem, and it's quite successful in doing so (compared to vanilla RNN). All the operations performed inside of LSTM can also be explained just as in CNN.

There're many novel architectures researched, but the widely used ones are classical feedforward networks, convolutional networks and recurrent networks (LSTM / GRU).

For example when the data is NOT continuous

Text is obviously not continuous. The way you use NNs for text is you represent each word using one-hot-encoding and then apply fully connected layer to the input. Essentially, it'll learn word embeddings (representations) that're useful for your problem.