# non linear neural network, how?

I have successfully created a linear neural network in tflearn:

import numpy as np
import tflearn

X = [[0, 0, 1],
[0, 1, 1],
[1, 0, 1],
[1, 1, 1]]

Y = [[0, 1],
[0, 1],
[1, 0],
[1, 0]]

Xtest = np.array([[1, 1, 0],
[0, 1, 0],
[1, 0, 1],
[0, 1, 1]])

# Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net)

# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(X, Y, n_epoch=10000, batch_size=16, show_metric=True)

pred = model.predict(Xtest)
for i in range(4):
print(pred[i][0])


the prediction for Xtest gives the correct answer: [1, 0, 1, 0]

Training Step: 10000  | total loss: 0.22091 | time: 0.002s
| Adam | epoch: 10000 | loss: 0.22091 - acc: 0.9491 -- iter: 4/4
--
0.9961615800857544
0.049626611173152924
0.9893295168876648
0.009963239543139935


However, when i make the matrix NON-linear:

X = [[0, 0, 1],
[0, 1, 1],
[1, 0, 1],
[1, 1, 1]]

Y = [[0, 1],
[1, 0],
[1, 0],
[0, 1]]


my prediction accuracy becomes 50/50 (because the matrix is no longer a linear relationship) and the predicted answer is wrong:

Training Step: 10000  | total loss: 0.69315 | time: 0.002s
| Adam | epoch: 10000 | loss: 0.69315 - acc: 0.5032 -- iter: 4/4
--
0.501151978969574
0.5012017488479614
0.49844005703926086
0.5015537738800049


how do i set this up correctly for the non-linear relationship?

You need to apply an activation function after your hidden layers, otherwise your network is just the softmax of a linear transformation.

For example, replace

net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 32)


with

net = tflearn.fully_connected(net, 32, activation='relu')

• thanks, i tried that but im getting a max accuracy of 75% now Commented Jun 4, 2017 at 7:19
• @Mark Markrowave Charlton works fine for me Commented Jun 4, 2017 at 7:28
• can you send me your full code? what accuracy are you getting? Commented Jun 4, 2017 at 7:31
• I literally just made the change in my answer, but almost any architecture will work for such an easy problem. I am getting 99.99-100% accuracy. Commented Jun 4, 2017 at 7:35
• oh my goodness i must of messed something up, its working just fine now thanks so much for your help! Commented Jun 4, 2017 at 7:36