# r neuralnet forcing overfitting

I am trying to get a neural network (neuralnet in R) to fit this function with a squiggly bit (black line).

No matter how I set up my neural network, it seems the fit (red line) always finds the overall trend while ignoring the squiggle. It's not often that lack of overfitting is the problem!

I've tried setting the network to various sizes and depths, from 4 to 40,20 or 9,9,9,9, and various other options in the neuralnet package have been fiddled with.

Here's the code I used to generate this, if anyone is interested:

library(neuralnet)
x<-seq(0,1,by=0.001)
yfunct<-function(x){
sqrt(x)+0.07*sin(25*x)*dnorm(x,mean=0.6,sd=0.1) #the hidden function
}
y<-yfunct(x)
nn<-try(neuralnet(y~x,data=dff,linear.output=FALSE,hidden=c(9,9,9,9),lifesign="full",threshold=0.01,stepmax=100000))

nny<-compute(nn,seq(0,1,by=0.001))
plot(x,y,type="l")
lines(x,nny$net.result,col="2") sd(nny$net.result-y)

• what is dff? a data frame containing x and y? – P.Windridge Sep 12 '17 at 12:59

With a small typo correction to your code, I get

This is simply by inserting dff <- data.frame(x=x,y=y) before the call to neuralnet.

After playing around though I suspect that's not your issue. Possibly you have hit a local near-optimum that is triggering the convergence criteria. Moreover, I think the problem is exacerbated by the "continuous" nature of your training data.

I can get some improvement on your picture using a small single hidden layer, like

dff <- data.frame(x=x,y=y)
set.seed(120917)
nn<-neuralnet(y~x,data=dff,linear.output=FALSE,hidden=c(3),lifesign="full",threshold=0.01,stepmax=5000) #, act.fct = "tanh")
nny<-compute(nn,seq(0,1,by=0.001))
plot(x,y,type="l")
lines(x,nny$net.result,col="red")  On the other hand, adding an extra layer and keeping the same threshold on the derivatives for convergence, i.e. set.seed(120917) nn<-neuralnet(y~x, data=dff,linear.output=FALSE,hidden=c(3,3),lifesign="full", threshold=0.01, stepmax=20000) #, act.fct = "tanh") nny<-compute(nn,x) plot(x,y,type="l") lines(x,nny$net.result,col="red")


gives

Keeping everything the same but with a big random perturbation for the weights gives something reasonable even with 2 layers:

# shuffle weights and start again
startweights<-nn$weights for (i in 1:3) { wts <- startweights[[1]][[i]] startweights[[1]][[i]] <- wts + runif(n=length(wts),min=-10,max=+10) } nn<-neuralnet(y~x, data=dff,linear.output=FALSE, hidden=c(3,3),lifesign="full", threshold=0.01, stepmax=20000, err.fct = "sse", startweights = startweights) nny<-compute(nn,x) plot(x,y,type="l") lines(x,nny$net.result,col="red")


I suspect it's because the derivative in each parameter becomes small rather quickly if you hit a local optimum like a smoothed or "mean" target. (There was a recent question concerning what I think was the same phenomenon). I do not think you want to try overfitting in response to this but rather look at starting values, convergence criteria, error function or algorithm (listed in increasing difficulty of investigation).