1
$\begingroup$

I have to extract a specific curve from an aggregate curve. An example is illustrated in the picture below, the blue curve is the aggregate curve, and I want to extract the red curve from the latter.

Blue: aggregate curve, Red: curve to be extracted

The idea is to use a neural network that takes the blue curve as input and generates the red curve as output. I am trying to perform some tests with feedforward neural networks (with Tensorflow library) but the results are very bad (the accuracy continues to decrease over the training). I suppose that the problem is the loss-function, I tried the cross-entropy with sigmoid (tf.nn.sigmoid_cross_entropy_with_logits()) and the cross-entropy with softmax (tf.nn.softmax_cross_entropy_with_logits()). Which loss-function would be more appropriate for this kind of problem?

Update

Network architecture:

  • Input: a sequence of 48 float32 values (the blue curve).
  • Output: a sequence of 48 float32 values (the red curve).
  • Optimizer: Adam (tf.train.AdamOptimizer()).
  • Batches of 100 samples.
  • Layer 1: fully connected with 96 neurons.
  • Layer 2: fully connected with 96 neurons.
  • Layer 3: fully connected with 48 neurons.

I calculate accuracy as: 1 / (1 + abs(output - correct_output) / correct_output)

Generally I start with an accuracy of around 50%, and after 50'000 training steps it is around 48-47%.

Improve this question
$\endgroup$
2
  • $\begingroup$ First tell us what is the architecture of the network you are trying to use and the accuracy you are getting. Did you try increase or decrease number of neurons in hidden layer and increase/decrease number of hidden layers. $\endgroup$
    – papabiceps
    Commented Jul 13, 2017 at 11:20
  • $\begingroup$ Yes, I have just added more information, thanks for you comment. And yes, I tried to add more hidden layers and to increase the neurons of them. Sorry for the late. $\endgroup$
    – Nicola
    Commented Jul 14, 2017 at 8:34

1 Answer 1

Reset to default
1
$\begingroup$

Why are you using a classification loss? sigmoid with BCE and softmax with BCE are loss functions used for classification problems. Your problem is not a classification problem, as far as I can see. It is a kind of a denoising problem. You need to use the appropriate loss function, which would be the l2 loss or the l1 loss. I would go with the l2 loss first (tf.nn.l2_loss). I would look into the l2 loss between the ground truth red curve and the output of the network.

As far as what you have written, your architecture is OK. two hidden layers are probably enough to do this task. But do not use a non-linearity (such as tanh) at the output, because your data is in scale of 0 to 1000. In the worst case use ReLU, but I would just keep the output as it is.

I hope you have enough training data.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you! The L2 loss did not work, instead with L1 loss the neural network increases its accuracy, but the training is slow. With 1'000'000 training steps (batch of 100 samples) I obtained an accuracy of about 81-82%. Do you have some suggestion? $\endgroup$
    – Nicola
    Commented Jul 14, 2017 at 12:03
  • $\begingroup$ i) It is very weird, that it did not work with the l2 loss, it actually should if it works wit the l1 loss... There might be a bug. ii) I don't quite understand your loss metric accuracy. In the formula you use, are those vectors? Do you sum over the vectors and the batch? Your problem is a regression problem, not a classification, so can you instead use the l2 loss to report the error as well? iii) You report the error on the test dataset , right? You need to have seperate training and test datasets. $\endgroup$
    – user166243
    Commented Jul 14, 2017 at 12:17
  • $\begingroup$ can you post an example set with your test data: i) a single input curve, ii) its corresponding groudtruth disaggregated curve and ii) the output of the network to this input. Can you also report the l2 loss between the output and the groundtruth for that curve? $\endgroup$
    – user166243
    Commented Jul 14, 2017 at 12:22
  • $\begingroup$ i) I'm performing another test with L2 with 1e6 steps right now, I will post here the results soon. ii) I used this formula in python to compute the accuracy: 'accuracy = tf.reduce_mean(tf.reduce_mean((1 / (1 + (tf.abs(y - y_) / (y_ + 1e-12)))), 1))', where y is the network output and y_ is the desired output (they are lists with shape [100,48]). iii) For the error with L1 see here: imgur.com/h1KWyv0 $\endgroup$
    – Nicola
    Commented Jul 14, 2017 at 12:26
  • $\begingroup$ how do you implement the l1 loss? how many training data do you have? 1e6 steps seem to be a lot... what is your learning rate for the adam? $\endgroup$
    – user166243
    Commented Jul 14, 2017 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.