0
$\begingroup$

I am attempting to create a neural network model using R that estimates the sine function for a given x variable, i.e. sin(x). Dataset has 5000 observations with x iterating by 0.05 each time:

dataset

However, when I estimate using the model, my error is quite high at 219.2 and my prediction remains fairly constant at a value of 0.48 or thereabouts, while the actual value deviates sharply between 0 and 1:

neuralnetresults

The steps I am taking to estimate the model are as follows:

1) I am scaling my sin and x variables using max-min normalization:

sinscaled=(sin-min(sin))/(max(sin)-min(sin))
xscaled=(x-min(x))/(max(x)-min(x))

2) I am then fitting my training and test data:

trainset <- maxmindf[1:3500, ]
testset <- maxmindf[3501:5000, ]

3) The neural network is fitted as below, and the resulting output tested:

#Neural Network
library(neuralnet)
sinscalednet <- neuralnet(sinscaled ~ xscaled, data=trainset, 
hidden=c(2,1), linear.output=FALSE, threshold=0.01)
sinscalednet$result.matrix
plot(sinscalednet)

Note that while I am using 2 hidden layers in this instance, the error is remaining consistent even when the number of hidden layers are being manipulated:

resultmatrix

4) Resulting output is tested and accuracy obtained:

#Test the resulting output
temp_test <- subset(testset, select = c("xscaled"))
head(temp_test)
sinscalednet.results <- compute(sinscalednet, temp_test)

#Accuracy
results <- data.frame(actual = testset$sin, prediction = 
sinscalednet.results$net.result)
results
roundedresults<-sapply(results,round,digits=0)
roundedresultsdf=data.frame(roundedresults)
attach(roundedresultsdf)
table(actual,prediction)

confusionmatrix

Would be grateful for any advice on steps I need to take to improve this model. I am experimenting by trying to modify the hidden layers, but the error is remaining consistently high.

$\endgroup$
  • $\begingroup$ did you scale your training and test inputs together? Because if so, then $x_i > x_j$ $\forall i \in T_e$, $j \in T_r$ where $T_e$ is the test index set and $T_r$ is the training index set, so you're predicting on data well beyond the range of anything seen in training. You should randomly partition your data into training and test sets instead. $\endgroup$ – jld May 17 '17 at 21:32
  • $\begingroup$ Quite possibly - an oversight on my part. Will check and run again. $\endgroup$ – Michael Grogan May 17 '17 at 23:55
1
$\begingroup$

What @Chaconne mentioned in the comments is quite important. You should first shuffle your training set and then split the array into chunks.

But I rewrote your code to the Javascript neural network library Neataptic, and i'm not having any issues. This is the code I wrote:

// Create the network
var network = new Architect.Perceptron(1,2,1);

// Create training set
var set = [];
for(var i = 1; i <= 5000; i++){
  var x = i * 0.05;
  x = x / (5000 * 0.05);

  var y = Math.sin(x);
  y = (y + 1) / 2;

  set.push({ input: [x], output: [y]});
}

var trainingSet = set.splice(0, 3500);
var testSet = set;

// Train the network
var training = network.train(trainingSet, {
  iterations: 500, // it will never reach this
  error: 0.0001,
  rate: 0.01,
  cost: Methods.Cost.MSE
});

var testError = network.test(testSet, Methods.Cost.MSE);

Run the code yourself here (open console)!

I'm getting these kind of results somewhat consistently:

training error: 0.00008
test error:  0.00133

So dropout would fix the large gap, but my point is, something else must be wrong in your model. Consistently getting 0.48 as prediction seems like the network has never learned anything.

What is your training error? What is your test error? How long are you training for (iterations).

$\endgroup$
  • $\begingroup$ I have rearranged the data, but still get the same error. Moreover, when I run a neuralnet for an equal number of observations (for both training and test data), I get an error of 157 for 24 steps, and an error of 155 for 32 steps. Yes, something is wrong in the model but I can't quite work out what. In other neural network models I've run, I haven't encountered this problem. Even setting linear.output = TRUE has not alleviated the issue. $\endgroup$ – Michael Grogan May 18 '17 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.