Currently I am working on a simple, single hidden layer, neural network to estimate the parameters of a certain phenomenon. The network requires two inputs and estimates the single parameter.

The data set used for training is obtained from experiments. This data set contains two input variables and a single output/measurement. The inputs are clean whereas the output/measurement is known to be noisy. Goal for the neural network therefore is to model the surface of the output with respect to the inputs.

Part of this data set is obtained through different experiments and has very little (say, negligible) noise. This part of the data set is clustered at the centre of the measurements and accounts for ~1% of all data points. Furthermore, the density of data points is much less in this patch than it is outside of the patch.

My question, how should I treat this patch with data which essentially yield more information about the true behaviour of the data.

Thanks in advance!


1 Answer 1


You may modify the cost function used during training and penalize less when the neural network does not precisely predict a noisy output. However, if all that you know is that the variance of noise is $0.08$ in the noisy cluster, and noiseless data make only $1\%$ of all the examples, I don't think that accounting for noise will be significantly useful, but it is certainly worth trying, at least for curiosity. :)

Let $\overline{\varepsilon}$ denote the average noise and $\overline{Y}$ stand for the average value of the actual response. Let $Q^{(i)}$ represent the computed response of the $i$th input, and $Y^{(i)}$ denote the calculated response. If $\overline{\varepsilon}/\overline{Y}$ is close to zero, noise is negligible and I would not bother with redesigning the network or the training algorithm. If the ratio is significantly different than $0$, exploiting noise may be a good approach.

For example, if $\varepsilon^{(i)}$ is the (estimated) value of noise in the $i$th response, you can add $(Q^{(i)}-Y^{(i)})^2$ to the cost function if $Q^{(i)} \notin [Y^{(i)}-\varepsilon^{(i)},Y^{(i)}+\varepsilon^{(i)}]$. This way you penalize predictions only if they are not in the interval of values derived with noise. But if $\varepsilon^{(i)}$ is not known, whereas the distribution of noise is known, the only way that I can think of right now is to draw noises from the distribution, assign them to the noisy responses and use the modified cost function. You may draw several sets of noise and validate them on a separate set before testing. But this may be too complex and time-consuming whereas it is questionable if the network will profit from the approach.

  • $\begingroup$ That sounds like a very interesting option! Can you please elaborate a little more on why your method could improve the estimate of the NN? As for your solution, it is known that the noise of the samples outside of the noiseless patch have a certain (fixed) variance of 0.08. Would you then suggest to construct an input vector (x_3) with the value of 0.08 on all noisy data points and a value of 0 in the noiseless data points? Thanks! $\endgroup$
    – neefsteef
    Oct 9, 2016 at 22:09
  • $\begingroup$ Small remark, the noise is only present in the output of the data, so there is no noise present in x1 and x2. R $\endgroup$
    – neefsteef
    Oct 9, 2016 at 22:28
  • $\begingroup$ Oh, in that case, I can suggest another way. Instead of including noise to input, you could slightly change the cost function for training in a way that distance between computed and actual noisy output is penalized less than if the actual output is noiseless. I will try to come up with concrete formulae tomorrow, if you do not figure them out by yourself by then. $\endgroup$
    – Milos
    Oct 10, 2016 at 1:21
  • $\begingroup$ Intuition behind including noise to input is that you are feeding your neural network with additional information and so enabling it to better fit your data. You are not just giving it the data to work with, you are also providing information on how reliable that data is. I would prefer this way if noise was present in input data. If you can, edit the original question so that it is clear where the noise is. I have to admit that I understood it different than what you really meant. :) $\endgroup$
    – Milos
    Oct 10, 2016 at 1:26
  • $\begingroup$ Thanks for the elaboration, this sounds really nice. I also edited my question to stat that the output is noisy and the input is not. $\endgroup$
    – neefsteef
    Oct 10, 2016 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.