0
$\begingroup$

I have $\mathcal N$ random binary variables $\in \{1, -1\}$ and a data-set comprised of some possible combinations of these variables. I'm trying to train a neural network to find possible dependencies between these variables.

The problem with my training data-set is that each combination of the variables is not full, it contains only some of the variables. Exact value of each unspecified variable in each combination is unknown in the meaning that the variable wasn't physically measured when the data-set combination is collected.

As for now I’ve created a network with $\mathcal N$ inputs and $\mathcal N$ outputs. For each input variable I put $1$, $-1$ or $0$ for the case if input value is unknown. I perform a "dry run" (just feed forward without back propagation) to see what’s predicted for each unknown output. After that I create a training sample with all unknown outputs replaced with predictions from the "dry run" and perform a training run with back propagation. This is done to not to "disturb" predictions of the unknown output.

I can see that at least for small data-sets the network performs not quite well. So I'm not quite sure that my approach with "dry runs" is correct and more efficient than just putting $0$ as expected value for unknown outputs. Are there any concerns about correctness of my approach?

$\endgroup$
2
$\begingroup$

It seems what you're trying to do is building an autoencoder (see here), although I'm not sure what you mean by "dry" runs, I suspect you're just propagating the outputs as in normal backprop.

What you could try, rather than setting your variables to zero when missing, is to create dummies from your variables (ie for an $X \in \{-1,1\}$, create two variables $X_1, X_{-1}$ with $X_{1} = 1$ if $X=1$, $0$ otherwise, and similarly with $X_{-1}$. Then missing values are simply the case where both are zero. This will increase dimensionality (doubles the number of variables), but as long as you are keeping a similar size for your hidden layer(s), it shouldn't matter.

You could also try taking a look at Restricted Boltzmann Machines (see here), which do a similar job for binary variables.

$\endgroup$
  • $\begingroup$ By "dry" run I mean just feed forward to peek predictions and use the predictions for backprop instead of unknown outputs. Dimensionality of input layer is not the main concern here. The real concern is that I don't know correct outputs for specified input. If I encode the unknown outputs with two variables as proposed, the net would learn to predict whether the output is unknown rather than its real value. So the solution doesn't quite applicable for my case. $\endgroup$ – Volodymyr Frolov Dec 22 '15 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.