1
$\begingroup$

I have been working on a neural network based predictor for a project. The aim is to learn a certain quantity, say the signal strength of a cellular network, for each coordinate set in the dataset. There is a discernible pattern to the data which I know from plotting it i.e. high signal strength near a tower and low further away. However, the output of the network seems to struggle to go beyond a certain maximum and minimum value.

Here are some details of what I have tried to improve this and my overall design:

  • I am using a two layer feed forward fully connected NN.
  • The implementation is in Keras using a tensorflow backend.
  • The input coordinates have been rescaled to between 0 and 1 and I have tried a cartesian input conversion too.
  • The two layers have sigmoid activations and the ouput layer is linear.
  • I have trained the network extensively, up to 10k epochs and the results are pretty much the same.
  • There is literature already showing that modeling signal strength is possible with this set of inputs

In my experience, an output limiting itself is due to the activation being oversaturated, often due to the input not being scaled. However, I think I have catered to that reasonably well. Changing the number of neurons, additional layers, and even activations (relu, linear, tanh) didn't help. And I am using the Adam optimizer with default values.

$\endgroup$
  • $\begingroup$ how about rescaling the output to a reasonable range? $\endgroup$ – shimao Aug 20 '18 at 10:17
  • $\begingroup$ I tried that too. It has no effect on the results $\endgroup$ – malesam Aug 20 '18 at 12:53
  • $\begingroup$ What kind of loss function are you using? If you are using a mean square error loss for example, you neural network might have learned that it is risky (i.e. very large loss) to predict very large or very small values as it hasn't found a good feature to predict them accurately. $\endgroup$ – GR4 Aug 24 '18 at 13:53
  • $\begingroup$ I have tried a bunch of loss functions, and mean absolute error tends to work the best. And yes, this behavior was made worse using mean square error. $\endgroup$ – malesam Aug 25 '18 at 1:39
0
$\begingroup$

I ended up rephrasing the entire problem as a time series prediction modeled with a recurrent NN with LSTM cells. This yielded significantly better results. I think that the problem as I was phrasing it originally was beyond the learning abilities of a simple feed-forward neural network and it was trying to converge to the mean

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.