1
$\begingroup$

I'm writing my own implementation of a neural network to test my knowledge, and while it seems to run okay, it converges such that the output is always the mean value (0.5 since I'm using logistic output activation) regardless of the input, and nothing I do seems to change anything. There are a few similar questions on, but I've tried the suggestions there to no avail.

So far I've tried the network on an XOR gate, and on the function f(a + b) = c for a, b in the range [0, 0.5) using a squared difference loss function. Both of these networks have two inputs, one output, and I've tried a number of architectures, including 1 and 2 hidden layers with about 3-8 nodes in each. All of the neurons are using a logistic activation. Stochastic, mini-batch, with bias/without bias, all seemed to make no difference; same for the learning rate, for which I've done a number of values between 0 and 1. The only thing I can think of it that it's the initialisation (I'm currently initialising weights according to a uniform distribution with small min/max centred about 0) but wouldn't know how to fix that even if it were wrong.

I have looked at the deltas I'm getting for the weights and they seem sensible. After training for a while the weights look fairly sensible too, most between -1 and 1 though there are sometimes a few that are a lot larger. Training error starts off relatively high, then comes down within a few epochs and stays there, sometimes even increases slightly. Any help would be greatly appreciated.

$\endgroup$
  • $\begingroup$ Have you tried replicating your net and training with e.g. mxnet? Based on my (very) limited experience even a small one hidden layer should perform better than yours but it's difficult to say there's a bug in your training (or maybe prediction) code without more detail. $\endgroup$ – P.Windridge Sep 10 '17 at 20:21
  • $\begingroup$ I haven't yet but that's the next port of call. I did test the training code as I wrote it but I'm just going through now and testing it more thoroughly. $\endgroup$ – QuadraticOne Sep 10 '17 at 20:30
  • $\begingroup$ In order to solve XOR problem with a shallow neural network, you have to iterate quite a lot. How many epochs have you tried? $\endgroup$ – Buomsoo Kim Sep 11 '17 at 8:34
  • $\begingroup$ I did a fair few iterations (can't remember the exact number) but have since changed my implementation round a bit and it now converges nicely, well within 50k epochs (on just the four training points) or so. $\endgroup$ – QuadraticOne Sep 11 '17 at 22:33
1
$\begingroup$

[An extended comment, rather than an "answer", which I think would be difficult from the information provided]

I tried to quickly replicate the scenario you're looking at with mxnet. The result was accuracy > 98%, with a decent spread in output. Histogram of logistic output vs target variable below. I also tried targeting RMSE and still get good results. Apologies if I misunderstood your description of the test case data!

library(mxnet)

x <- data.frame( a= runif(500), b= runif(500))
y <- as.numeric((x$a <0.5) * (x$b < 0.5))
x <- data.matrix(x, rownames.force = NA)

model <- mx.mlp(x, y, hidden_node=5, out_node=2, out_activation="softmax",
                num.round=100, array.batch.size=15, learning.rate=0.07, momentum=0.9,
                eval.metric=mx.metric.accuracy) # mx.metric.rmse, mx.metric.accuracy)

Histogram of predicted

Update: I played more and found training the network setup below to gets stuck at something that predicts near constant (generate data above with set.seed(0) too). Thus perhaps there is some phenomenon happening here that someone more experienced can comment on.

set.seed(0)
model <- mx.mlp(x, y, hidden_node=c(3,3), out_node=2, out_activation="softmax", activation="tanh",
                num.round=50, array.batch.size=15, learning.rate=0.07, momentum=0.9,
                eval.metric=mx.metric.accuracy) # mx.metric.rmse, mx.metric.accuracy)

With initializer = mx.init.uniform(1) (i.e. uniform[-1,1] values this seems to go away). Generally with 2 layers there seem too many degrees of freedom.

$\endgroup$
  • $\begingroup$ That's encouraging! I guess it must be something wrong with my implementation... Only major difference is that I used a single output with sigmoid activation rather than softmax, and also no momentum, but I don't think that explains such a huge difference in output. Thanks very much :) $\endgroup$ – QuadraticOne Sep 10 '17 at 21:46
  • $\begingroup$ You're welcome! After playing more I also observed an instance where the network appeared to stick on constant predictions, although I didn't do many iterations. With all the examples I found that using a bigger range for the uniform starting values seem to avoid the problem $\endgroup$ – P.Windridge Sep 10 '17 at 21:56
  • $\begingroup$ So I went through a load of unit tests to see where it was going wrong, turns out it was in my implementation. It may or may not have been a typo in my matrix multiplication method... Anyway, I learnt my lesson. I've put in the hyperparameters in that you used and it works beautifully. Thanks again for your help! $\endgroup$ – QuadraticOne Sep 10 '17 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.