I have a dataset containing 30 million rows and 500 columns. A simple 32x32 MLP appears to overfit the data, in that the training error goes down nicely but the test error goes up. Given that my training set is so large, I suspect the overfitting is occurring because my training examples are too similar to each other. How would I go about confirming this? e.g. what techniques are out there for finding similar examples or assessing the variance of my training set? Or how would I increase the heterogeneity of the dataset?
I should say that my data was captured as a time series, so adjacent examples would naturally be quite similar if nothing happened between the two timestamps, but each training example has context, i.e. contains information about what happened in a certain time window around the current time. Right now I'm shuffling the data and trying to do classification.