1
$\begingroup$

I am trying to modify Andrej Karpathy's char-RNN code. As far as I understand, the loss function used in his code for a LSTM is the Softmax function function (in the file model/LSTM.lua ). I understand Softmax is the multi-class equivalent of the Logistic loss function (used for 2-class classification).

The site here says that the Softmax function must have only ONE of its inputs high (i.e, output of the network should be 1-hot vector before applying softmax). Softmax works with 1-hot input only

I want to modify the code to train an LSTM which gives a real valued vector (i.e, if there are C classes, a C-dimensional vector of floats) and then apply an appropriate loss function on top of it.

My question is, which loss functions are suitable for real-valued vector inputs and how to modify the LSTM in char-RNN implementation to achieve it? (It would be helpful if such a function already exists in the lua/torch libraries. If it doesn't, it would be really helpful if you could provide me the appropiate lua code and integrate it with the LSTM.lua file. I am new to Torch/Lua programming and Deep Learning as such and would really appreciate you providing your precious time to help me. Please forgive me if some of my queries seem foolish.)

$\endgroup$
4
  • $\begingroup$ Note that code questions are generally off topic here, so you may not get that, but the main part of your question should be answerable here. $\endgroup$ Commented May 18, 2016 at 19:27
  • $\begingroup$ I think you are confusing a couple of things. First of all, the loss function defines how happy you are with an arbitrary (prediction, target) pair. It has nothing to do with the (type of) input variables. $\endgroup$
    – Pieter
    Commented May 18, 2016 at 19:52
  • $\begingroup$ Secondly, a softmax layer is a type of layer, just like a linear, sigmoid or pooling layer. It takes care of normalizing the activation values of all the nodes in the layer such that the sum is equal to 1. This means that it can be interpreted probabilistically. $\endgroup$
    – Pieter
    Commented May 18, 2016 at 19:52
  • $\begingroup$ @Pieter , I understand. By input, I meant input to the function (the prediction of the network). $\endgroup$ Commented May 18, 2016 at 19:55

1 Answer 1

2
$\begingroup$

If you have a continuous output that is not restricted to a certain interval, e.g. in a regression problem, you could use the regression criterions. Often the mean squared error (MSE) is used which is also implemented in Torch/Lua.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.