2
$\begingroup$

This is the most widely used method for outlier detection in econometrics and statistical problems. X is our data that we're searching for outliers in it (in MATLAB) :

 abs(X-mean(X)) >= n*std(X)

So if this inequality was true, that sample is an outlier; otherwise we will keep the sample. I'm using neural network and SVM for my classification problem. Before normalization of data, I find outlier for each feature in input database. After creating my MLP or SVM model now I want use it for out-sample data ( new data from this year - model trained by data of last year ).

  • Should I use outlier detection in this new database?
  • If answer to that question is yes, Should I use mean and std of previous year that was used for outlier detection of main data, or use new mean and std of out-sample data?
$\endgroup$
4
  • 1
    $\begingroup$ Given you flag a data-point as an outlier because it does not conform with some criterion A, why would you try to judge the goodness of your model, which was designed/trained using data that follow A, to estimate against a value that does not follow that criterion? You just said it was nonsensical (based on criterion A). On the same manner, the out-sample data, are out-sample data. The statistics of your training sample should not be applied to your test sample. Otherwise you are re-using your data. $\endgroup$
    – usεr11852
    Aug 14, 2014 at 21:52
  • 1
    $\begingroup$ Thank you for answer. But when we use normalization in creating system, we use max,min,std and mean of data for normalization of out-sample data. because of that I thought maybe we have same method in outlier detection. What do you think? So I should have outlier detection in out-sample data (with mean and std of out-sample) or don't have outlier detection in out-sample data? ( my out-sample data is not test sample - That is a new database that should check with destined model - this year data that we need prediction of it) $\endgroup$ Aug 14, 2014 at 21:54
  • $\begingroup$ Please refrain from repeatedly making tiny edits to your question, this is very much frowned upon here, and you might end up not getting any answers at all. It is completely legitimate and even a good idea to edit your question to "bump it up" to the front page again, but it should be substantial edits, showing continuous research effort from your side (or providing important clarifications about your problem). $\endgroup$
    – amoeba
    Aug 15, 2014 at 16:53
  • $\begingroup$ @amoeba . Sorry. I didn't know about that. Thank you for your comment amoeba. $\endgroup$ Aug 15, 2014 at 16:56

1 Answer 1

1
$\begingroup$

The rule for outlier detection you defined before training your model should be applied to all new data you will use, otherwise your model will be confronted with data points it has not be trained to handle. When you apply this rule it should be the exact same rule, i.e. it should use the mean and std from the previous year.

However if you expect significant difference between the means of two successive years this might result in flagging too many data points as outliers. You can check that quickly by looking at the moving average.

Regarding the outlier detection, using one feature at the time might not be optimal, especially if you use multivariate methods afterwards. Some interesting tools re available here in different implementations (MATLAB, R ...).

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. I have second paragraph problem. When I insert new database to my model, the number of outliers is so high (about 30%). How can I solve this? $\endgroup$ Aug 20, 2014 at 8:48
  • $\begingroup$ If you believe that 30% is too high, then it might be that your rule for outlier detection is too stringent. Relax your rule (changing n in your formula). $\endgroup$
    – LionelB
    Aug 22, 2014 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.