0
$\begingroup$

I am giving a hypothetical example to convey my question. Suppose I want to train a neural network that abbreviates strings with a preset list of words that are likely to be present in full form of the word. So

database = db

because

data and base are words. We can definitely train a network that can be trained to learn this kind of mapping given large training set with enough instances for each word. But can I train a general neural network which can work with a novel set of words with pre-trained network

In my example, I should be able to train a network which once trained can accept a new set of words that might appear in the examples without a need for retraining with those words.

I am thinking something along the lines of NN working with support data which could be changed as per needs.

$\endgroup$
0
$\begingroup$

I think what you're saying is, you want your network inputs to be independent of the words in your training set. So you don't want to rely on having "data" and "base" in your dictionary and then splitting "database" into "data" + "base".

One approach would be a character-level LSTM, which ingests your input word one character at a time, and then after each character, makes a prediction of "(blank)", or "letter", so that the final abbreviation is the concatenation of the predictions after removing (blank)s. For example with "database" it would potentially predict "d(blank)(blank)(blank)(blank)(blank)(blank)b" which becomes "db".

$\endgroup$
  • $\begingroup$ I am asking if it is possible to have a trained network change its behavior without training it with new examples? So your LSTM would work fine for database to db but will it work if I just know some new list of words but I don't want to retrain the network for those words? AFAIK it is not possible with normal NNs but maybe NNs could be made to work like turing machines. $\endgroup$ – amitamb Jun 19 '18 at 21:09
  • $\begingroup$ It's a character level lstm. So as long as you have examples of letters "a-z" in your training set, it will work on arbitrary words. $\endgroup$ – Alex R. Jun 19 '18 at 21:45
0
$\begingroup$

The whole point of neural networks is to be able to generalize. In other words, neural networks would be useless if they could only ever be used on your training data and will completely fail on test (real world) data. The training of a neural network is supervised training so presumably you already know all the answers for your training data. Hence, to your general question of "can a neural network work with (support) data which was not there while training" the answer is of course yes - depending on what you mean by "support" data.

The million dollar question is not can neural networks generalize (they certainly can), but will my neural network generalize to the extent that I want it to? There is no general answer to this question, it depends on the neural network and how you train it. There's a lot of issues that arise with this question including issues of overfitting and covariate shift. You would have to experiment to see if you can get a neural network to make predictions up to your standards.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.