How to train a neural network for regression when correct outputs aren't fully known?

Question

I have $\mathcal N$ random binary variables $\in \{1, -1\}$ and a data-set comprised of some possible combinations of these variables. I'm trying to train a neural network to find possible dependencies between these variables.

The problem with my training data-set is that each combination of the variables is not full, it contains only some of the variables. Exact value of each unspecified variable in each combination is unknown in the meaning that the variable wasn't physically measured when the data-set combination is collected.

As for now I’ve created a network with $\mathcal N$ inputs and $\mathcal N$ outputs. For each input variable I put $1$, $-1$ or $0$ for the case if input value is unknown. I perform a "dry run" (just feed forward without back propagation) to see what’s predicted for each unknown output. After that I create a training sample with all unknown outputs replaced with predictions from the "dry run" and perform a training run with back propagation. This is done to not to "disturb" predictions of the unknown output.

I can see that at least for small data-sets the network performs not quite well. So I'm not quite sure that my approach with "dry runs" is correct and more efficient than just putting $0$ as expected value for unknown outputs. Are there any concerns about correctness of my approach?

Answer 1

It seems what you're trying to do is building an autoencoder (see here), although I'm not sure what you mean by "dry" runs, I suspect you're just propagating the outputs as in normal backprop.

What you could try, rather than setting your variables to zero when missing, is to create dummies from your variables (ie for an $X \in \{-1,1\}$, create two variables $X_1, X_{-1}$ with $X_{1} = 1$ if $X=1$, $0$ otherwise, and similarly with $X_{-1}$. Then missing values are simply the case where both are zero. This will increase dimensionality (doubles the number of variables), but as long as you are keeping a similar size for your hidden layer(s), it shouldn't matter.

You could also try taking a look at Restricted Boltzmann Machines (see here), which do a similar job for binary variables.