Deep neural network regression with non-smooth data

Question

How does a deep neural network, trained for regression with back propagation, deal with cases when different training pairs have the same input value, but different output values, i.e. the relationship between input and output is not smooth?

In other words, suppose I train a deep neural network for regression from $x$ to $y$, using back propagation. Two of my data pairs have the same $x$ ($x_1 = a$ and $x_2 = a$), but very different $y$'s ($y_1 = b$ and $y_2 = c$). After training the network, what would be the output $y_k$ for an input of $x_k = a$? Would it be $b$, or the average of $b$ and $c$?

Answer 1

Have you trained some network with the setting you have mentioned? I have not yet, but if we think in some function like $y=\sqrt{x}$ or $y=sin^{-1}(x); x\in [0,2\pi ]$, these are simple functions, however if you want to approximate them, complex values should be supported by the learned transformation $x\mapsto f(x) = y$, i.e. in order to approximate functions like the mentioned ones, the network's architecture must be endowed with complex valued basis funtions 1. Otherwise, the imaginary part of the image of $f(x)$ could be lost. It is seemed to the case of approximating an injective function, where more than a $x$ corresponds to a $y$, e.g. $y = f(x) = x^2$. Then try to approximate $f^{-1}(y) = x$.

I hope this helps.

Complex valued neural networks