1
$\begingroup$

This problem has bothered me for a while now, and I'm not quite sure how to debug it.

I have a dataset comprised of graphs, i.e. each sample in the dataset is a graph. This is a regression task, but the number to predict is bounded between 0 and 1 (inclusive). The number of samples is in the thousands; I have small N because this is a biological dataset.

I am training a small graph convolutional neural network for this learning task. The number of model parameters often range from tens to hundreds at most; I have been reluctant to build models with larger numbers of parameters.

As this task is predicting a number between 0 and 1, my final layer has a logistic output, and I am using a mean squared error loss.

The problem I am facing is that regardless of model architecture, as long as I am training a model that has a modicum of depth, I seem to very quickly hit an empirical limit on my test loss.

My models are all able to overfit on small subsets of data, thus confirming that training (or more accurately, memorization) is happening. However, when I do a train/test split on a larger fraction (e.g. half to 3/4) of the dataset, I have empirically noticed that the loss on the test set will empirically bottom out at ~0.05 (MSE), implying an average absolute loss of 0.22 (which is horrible for this task: 22% prediction error on average). A more decent model ought to give 0.01 (MSE) (~10% prediction error on average).

By comparison, I have tried the following toy graph learning problems:

  1. Learn to count the number of nodes in a graph, given the underlying graph.
  2. Learn to sum up scalar (and vector) values attached to nodes on the graph.

In each of these scenarios, overfitting never happens; future data obeys the same rules as current data, i.e. they come from the same distribution.

Having written all of that, I am left baffled as to why this might be happening. Has anybody here faced this same issue and successfully resolved this? Could it be that the measurement error in the experiment that generated these data be right there at ~22%? Are there other potential problems that I have not described here that I should check for?

$\endgroup$
  • $\begingroup$ You mean graphs like in graph theory or plots? $\endgroup$ – Jakub Bartczuk Apr 18 '18 at 12:46
  • 1
    $\begingroup$ Do you use any regularization? Weight decay? Dropout? Batch normalization? $\endgroup$ – Jakub Bartczuk Apr 18 '18 at 12:47
  • $\begingroup$ @JakubBartczuk these are graph objects, i.e. networks. $\endgroup$ – ericmjl Apr 18 '18 at 12:59
  • $\begingroup$ I have experimented (but not systematically) with dropout regularization. These are standard methods that are on my list of things to try, but perhaps let me update my post with an additional observation that I made on some toy data. $\endgroup$ – ericmjl Apr 18 '18 at 13:00
  • $\begingroup$ The updated post now describes a few toy learning problems I tried. The short story is that overfitting never happens there, because future data and current data obey the same rules, or more generally, come from the same distribution. Could this be the thing that is missing from my current learning task? $\endgroup$ – ericmjl Apr 18 '18 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.