0
$\begingroup$

I have a large dataset (400k rows) in which I suspect the data has been obfuscated by the addition of a Gaussian distribution. My guess is that some of the data had categorical variables (based on the description of the data), or some other distribution within the data (e.g. bimodal).

For example, the data may look something like:

signal = np.random.randint(low=5, high=10, size=50000)
noise = np.random.normal(loc=8, scale=5, size=350000)
final_data = np.concatenate([signal, noise])

If it was possible to estimate the mean & standard deviation of the noise distribution (through some optimisation), how could I extract the signal from the noise?

EDIT: The spread of the noise distribution is wide enough to effectively hide the categorical data on visual inspection.

enter image description here

My initial idea was to compare the CDF of a perfect Gaussian against the data to look for differences but I'm not sure how to extract the signal using this method

$\endgroup$
0
$\begingroup$

If you know that the original data was integers, and if the noise added was suitably small, then you could simply round the new values and get the original values to a pretty good degree of accuracy. It would depend on the sd of the noise that was added. Then you could look at the difference between the recovered value and the original and find its mean and sd and also plot it to see if it is, in fact, Gaussian.

EDIT: If the noise added is large then I don't see a way to solve this. But adding noise that way is a bad idea. Why do you think it was done? Can you contact the authors?

$\endgroup$
  • $\begingroup$ Thanks Peter. I don't think my question was clear enough - I've added some extra details $\endgroup$ – Anjum Sayed Mar 17 at 11:11
  • $\begingroup$ Regarding the data source - the data is from a competition where transactions have to be predicted, and every feature looks almost Gaussian. I'm not certain that noise has been added, but there are no categorical features, and given the topic of the competition I would certainly expect some. My theory is that the organiser has intentionally obfuscated the data $\endgroup$ – Anjum Sayed Mar 17 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.