# Noisy data set with partially noiseless patch for simple neural network training

Currently I am working on a simple, single hidden layer, neural network to estimate the parameters of a certain phenomenon. The network requires two inputs and estimates the single parameter.

The data set used for training is obtained from experiments. This data set contains two input variables and a single output/measurement. The inputs are clean whereas the output/measurement is known to be noisy. Goal for the neural network therefore is to model the surface of the output with respect to the inputs.

Part of this data set is obtained through different experiments and has very little (say, negligible) noise. This part of the data set is clustered at the centre of the measurements and accounts for ~1% of all data points. Furthermore, the density of data points is much less in this patch than it is outside of the patch.

My question, how should I treat this patch with data which essentially yield more information about the true behaviour of the data.

You may modify the cost function used during training and penalize less when the neural network does not precisely predict a noisy output. However, if all that you know is that the variance of noise is $0.08$ in the noisy cluster, and noiseless data make only $1\%$ of all the examples, I don't think that accounting for noise will be significantly useful, but it is certainly worth trying, at least for curiosity. :)
Let $\overline{\varepsilon}$ denote the average noise and $\overline{Y}$ stand for the average value of the actual response. Let $Q^{(i)}$ represent the computed response of the $i$th input, and $Y^{(i)}$ denote the calculated response. If $\overline{\varepsilon}/\overline{Y}$ is close to zero, noise is negligible and I would not bother with redesigning the network or the training algorithm. If the ratio is significantly different than $0$, exploiting noise may be a good approach.
For example, if $\varepsilon^{(i)}$ is the (estimated) value of noise in the $i$th response, you can add $(Q^{(i)}-Y^{(i)})^2$ to the cost function if $Q^{(i)} \notin [Y^{(i)}-\varepsilon^{(i)},Y^{(i)}+\varepsilon^{(i)}]$. This way you penalize predictions only if they are not in the interval of values derived with noise. But if $\varepsilon^{(i)}$ is not known, whereas the distribution of noise is known, the only way that I can think of right now is to draw noises from the distribution, assign them to the noisy responses and use the modified cost function. You may draw several sets of noise and validate them on a separate set before testing. But this may be too complex and time-consuming whereas it is questionable if the network will profit from the approach.