This problem has bothered me for a while now, and I'm not quite sure how to debug it.
I have a dataset comprised of graphs, i.e. each sample in the dataset is a graph. This is a regression task, but the number to predict is bounded between 0 and 1 (inclusive). The number of samples is in the thousands; I have small N because this is a biological dataset.
I am training a small graph convolutional neural network for this learning task. The number of model parameters often range from tens to hundreds at most; I have been reluctant to build models with larger numbers of parameters.
As this task is predicting a number between 0 and 1, my final layer has a logistic output, and I am using a mean squared error loss.
The problem I am facing is that regardless of model architecture, as long as I am training a model that has a modicum of depth, I seem to very quickly hit an empirical limit on my test loss.
My models are all able to overfit on small subsets of data, thus confirming that training (or more accurately, memorization) is happening. However, when I do a train/test split on a larger fraction (e.g. half to 3/4) of the dataset, I have empirically noticed that the loss on the test set will empirically bottom out at ~0.05 (MSE), implying an average absolute loss of 0.22 (which is horrible for this task: 22% prediction error on average). A more decent model ought to give 0.01 (MSE) (~10% prediction error on average).
By comparison, I have tried the following toy graph learning problems:
- Learn to count the number of nodes in a graph, given the underlying graph.
- Learn to sum up scalar (and vector) values attached to nodes on the graph.
In each of these scenarios, overfitting never happens; future data obeys the same rules as current data, i.e. they come from the same distribution.
Having written all of that, I am left baffled as to why this might be happening. Has anybody here faced this same issue and successfully resolved this? Could it be that the measurement error in the experiment that generated these data be right there at ~22%? Are there other potential problems that I have not described here that I should check for?