I'm trying to find patterns in a large dataset using the neuralnet package.
My data file looks something like this (30,204,447 rows) :
id.company,EPS.or.Sales,FQ.or.FY,fiscal,date,value 000001,EPS,FY,2001,20020201,-5.520000 000001,SAL,FQ,2000,20020401,70.300003 000001,SAL,FY,2001,20020325,49.200001 000002,EPS,FQ,2008,20071009,-4.000000 000002,SAL,FY,2008,20071009,1.400000
I have split this initial file into four new files for annual/quarterly sales/EPS and it is on those files that I want to use neural networks to see if I can use the variables id.company, fiscal and date in the case below to predict the annual sales results.
To do so, I have written the following code:
dataset <- read.table("fy_sal_data.txt",header=T, sep="\t") #my file doesn't actually use comas as separators #extract training set and testing set trainset <- dataset[1:1000, ] testset <- dataset[1001:2000, ] #building the NN ann <- neuralnet(value ~ id.company + fiscal + date, trainset, hidden = 3, lifesign="minimal", threshold=0.01) #testing the output temp_test <- subset(testset, select=c("id.company", "fiscal", "date")) ann.results <- compute(ann, temp_test) #display the results cleanoutput <- cbind(testset$value, as.data.frame(ann.results$net.result)) colnames(cleanoutput) <- c("Expected Output", "NN Output") head(cleanoutput, 30)
Now my problem is that the compute function returns a constant answer no matter the inputs of the testing set.
Expected Output NN Output 1001 2006.500000 1417.796651 1002 2009.000000 1417.796651 1003 2006.500000 1417.796651 1004 2002.500000 1417.796651
I am very new to R and its neural networks packages but I have found online that some of the reasons for such results can be either:
an insufficient number of training examples (here I'm using a thousand ones but I've also tried using a million rows and the results were the same, only it took 4h to train)
or an error in the formula.
I am sure I'm doing something wrong but I can't seem to figure out what.