Why not start with one layer, and get the output equals the input for one layer? Then, by induction, you can just stack those one on top of the other, and just pipeline through your trained identity layers.
Note that if you're using ReLU, then if you have any negative values in your input/output, then ReLU will not be able to do the identity operation you seek. So you'd probably want to make sure your input/output is strictly positive.
Gradient vanishing refers to a specific effect of backpropagated gradients being attenuated by passage through an activation function, like tanh or sigmoid. ReLU doesnt suffer from vanishing gradients too much, since the gradient is 1, for positive domain input.
So:
- check your inputs/outputs are strictly positive
- start with a single layer
Edit: based on your new information that it works with a single layer, but not with multiple layers, I reckon that ReLU is blocking your gradient backprop, probably because it has zero gradient for much of its domain. Therefore, you might try using an activation function that has non-zero gradient almost everywhere, eg ELU or leaky ReLU. On the whole, I'd try leaky ReLU first, because:
- it is technically an activation function (cf not using an activation function at all)
- its piecewise linear (cf ELU)
- it has gradient almost everywhere, so gradients should backprop ok