6
$\begingroup$

I want to add some artificial outliers to my data set by follow same method below. so, how i can add contaminated data statistically to real data set like Pima Indians Diabetes?
info: Pima Indians Diabetes: 768 instances 8 attributes
enter link description here

Note: I want steps not programming code.

We have added uniformly distributed attributes as noisy attributes to data sets [15]. To compare outlier detection and false alarm rate in our experiments we have planted 3% to 5% artificial outliers into real data sets according to the data sets domain knowledge (statistical characteristics like mean, standard deviation, class distribution, type of attributes). Blockquote

enter link description here

$\endgroup$
5
  • $\begingroup$ Hello. Is this a statistics question (which would fit on this site) or a question about how to complete this task using certain software (which wouldn't fit)? $\endgroup$
    – rolando2
    Aug 23 '19 at 3:44
  • $\begingroup$ statistics question, i want steps to do statistically.ex: calculate mean or std $\endgroup$
    – user
    Aug 23 '19 at 3:50
  • $\begingroup$ You may have a different impression, but really there is no universally accepted definition of "outlier." Even the article you cite on outlier detection doesn't give much of a definition (based on a quick look); it talks about the need for outliers to meet certain criteria, and rather than specifying these it merely refers to another article ("13"). So it's not as if one could straightforwardly advise you create artificial values that are, e.g., >3.44 standard deviations from each variable's mean. You'll need to come up with your own criteria. Cheers ~ $\endgroup$
    – rolando2
    Aug 23 '19 at 4:06
  • $\begingroup$ stats.stackexchange.com/questions/7155/… $\endgroup$
    – rolando2
    Aug 23 '19 at 4:32
  • $\begingroup$ Thanks , how can i create attributes that is 3.3 standard deviations from each variable's mean? Take random sample from data set then add mean of data set+3.44 standard deviations of data set . $\endgroup$
    – user
    Aug 23 '19 at 4:42
6
$\begingroup$

One way would be to train a model that learns the distribution of each feature separately; it could be a KDE for each feature.

Then you could use this model to generate outliers for the data. I'd suggest producing the outliers by generating values at 4 std from the mean for a few of the features and generate realistic values for the rest. This will prevent the outliers to be too easily detectable.

$\endgroup$
4
  • $\begingroup$ Thanks.why this will prevent the outliers to be too easily detectable? . If i calculate each feature mean and std separately ,then take random features and add mean and 4 std? $\endgroup$
    – user
    Aug 24 '19 at 0:14
  • $\begingroup$ I think that the point should not be an outlier on all features. It's realistic if it is an outlier on one or two features alone. Yes that's what I'd do. $\endgroup$
    – kfn95
    Sep 1 '19 at 23:59
  • $\begingroup$ can do same rule for non normal distributed data? $\endgroup$
    – user
    Sep 2 '19 at 0:10
  • $\begingroup$ I think that the point should not be an outlier on all features? can explain more please .you mean if i have point with 4 features ,for example add outlier for 2 of them better than all 4 features? $\endgroup$
    – user
    Sep 2 '19 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.