I have an application where I get a large set of smooth curves (2D). Those curves are represented by sample points on that curve. Sometimes, those curves cross or get close to each other and it is not immediately clear which sample point corresponds to which curve.

Hence I try to generate a RNN that predicts the continuation of the curve. Like this, I can estimate, which sample point corresponds to which curve. An example curve

In the picture above you can see in blue the known curve and red the continuation. All curve are kind of sinusoidal, so no crazy non-smoothness.

The idea is to feed the RNN with the first n_samples known 2D points and let it estimate the next n_prediction points. Then find those points in the real data that are closest to those estimated points and use them to continue feeding the RNN until the curve has finished (the maximum number of sample points is known, e.g. say I have 10'000 points, then I feed e.g. 100 points and let the RNN predict the next 5 points. Then I find the closest 5 points and feed the RNN with the new curve)

I made this curve generator

class data_generator(object):
    def __init__(self, seq_length=50, shift=1, batch_size=1):
        self.seq_length = seq_length
        # shift is the number of points to be predicted
        self.shift = shift
        self.batch_size = batch_size

    def sample(self):
        # generate evenly spaced data pts
        offset = [np.random.uniform(0, np.pi*2), np.random.uniform(0, np.pi*2)]
        scalor = np.random.uniform(0.5, 1.5, (2,1))
        timor = 1+0.02*np.random.rand()
        time_steps = np.linspace(0, timor*np.pi, self.seq_length + self.shift)
        data = np.array([np.sin(scalor[0]*time_steps+offset[0]), np.sin(scalor[1]*time_steps+offset[1])])
        data.resize((2, self.seq_length + self.shift))

        x = data[:,:-self.shift]
        y = data[:,-self.shift:]
        x, y = x.T, y.T

        return x.reshape((1, x.shape[0], x.shape[1])), y

    def generate(self):
        while True:
            x = np.zeros((self.batch_size, self.seq_length, 2))
            y = np.zeros((self.batch_size, self.shift*2))
            for i in range(self.batch_size):
                _x, _y = self.sample()
                x[i,:, :] = _x
                y[i,:] = _y.reshape(1, -1)
            yield x, y

Then I create my LSTM net using keras

seq_length = 50
input_features = 2
shift = 2
output_features = input_features*shift
layers = 10
n_hidden = 32

model = Sequential()
model.add(LSTM(16, input_shape=(seq_length, input_features), return_sequences=True))
model.add(Dense(output_features, activation="relu"))
optimizer = Adadelta()

The output is just the 2D vector flattened to 1D. Later I would reshape that to a matrix with 2 columns (or 2 rows, the number of dimensions)

Then I try to fit the model

train_data_generator = data_generator(seq_length = seq_length, shift=shift, batch_size=20)
model.fit_generator(train_data_generator.generate(), steps_per_epoch = 16, epochs = 100)

x,y = train_data_generator.sample()
blub = model.predict(x)
pred = blub.reshape(-1, 2)
customplot(x.squeeze(0), y)
customplot(x.squeeze(0), pred)

My problem is, it does converge, but not satisfyingly. I tried different optimizer and different losses, but all in vain. Also the hyper parameters of the model itself I have changed, without real success. I feel this should work better, maybe I'm totally wrong?

The best I get (with mse loss) is a loss of about 0.27xx, which I believe is too large.

Any suggestions?

  • 1
    $\begingroup$ It’s not possible to conclusively answer this question without data. The only thing that stands out is your model output uses relu activations, which are necessarily nonnegative. Yet your curve clearly has negative coordinates. $\endgroup$ – Alex R. Jun 21 '19 at 17:09
  • $\begingroup$ Would you share your data? How does your dataset look like? Please show the excerpt of your dataset. $\endgroup$ – Mario Jun 24 '19 at 14:36
  • $\begingroup$ @Mario: It's all there, you can see how the data is being generated. $\endgroup$ – rst Jun 25 '19 at 12:54
  • $\begingroup$ @rst Did you notice that you have reshape problem for plotting? print(pred.shape)=(2, 2) print(y.shape)=(2, 2) print(x.squeeze(0).shape)=(50, 2) !! $\endgroup$ – Mario Jun 25 '19 at 14:59

Here is the general recipe: http://karpathy.github.io/2019/04/25/recipe/,

  1. You may specifically try to overfit one batch to check the model capacity.
  2. activation to Relu -> Tanh, also suggested in the comment above by Alex.
  3. Add another dense layer before the last layer to increase the capacity of the model to learn more nonlinearity of the curve Dense(4, activation='tanh')
  4. Also, you may try to increase the batch size to 35 assuming your seq_len=50. It worked for me when I was trying to train a simple autoencoder on sine and cosine waves.

Hope this helps. :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.