I am stuck in my data mining project. Actually, I don't know how to handle features that are not defined for some data. I try to explain better my situation. I am dealing with email data, and the problem is that some features are strictly related to email header field, so for example I extrapolate some informations from the Cc header field and, this header field is not always defined of course(and this is true for other fields as well). Now this means that the cell corresponding to this features for some data item will be missing, but actually it could not be considered as a missing value as it is not really something that is really missing. So I mean that it would not be correct to handle it using imputation. I though something like create multiple dataset based only on available features, but is there something already implemented about this in R or Weka tools? Or if someone know something better can please explain me how to solve this problem? Thanks in advance.

  • $\begingroup$ If my understanding is correct, empty cc header simply means that email has not cc'ed anyone. So it should definitely not be considered as missing. Could you provide more details on the format of the CC header field and how you would like to use it in your data mining model? $\endgroup$ – Bayesric Sep 4 '17 at 19:39
  • $\begingroup$ @Bayesric yes, it is as you understand. First of all thanks for your reply. For example one feature about Cc is about the fact if it is actually containing the value redacted, but if the field is not defined at all, considering it as a missing feature is, for my understanding, a wrong assumption. Another example is a feature about the Received-SPF field value, but again, if it is not defined it is wrong to perform some imputation procedure to estimate a value. It s like to deal with not mandatory features. And is true for a lot of other features. I hope to have been as clear as possible. $\endgroup$ – Myke Sep 4 '17 at 20:41
  • $\begingroup$ Why can't you just replace these "missing" values with some values that would probably not occur naturally in your dataset but are still valid inputs? Maybe you could use an empty string or a single space. Or you could define your own placeholder like "_&&&XXThisValueIsMissingSoIReplacedItXX&&&_". Would that work for what you are doing with the values later? $\endgroup$ – Secespitus Sep 5 '17 at 9:46
  • $\begingroup$ @Secespitus Actually I though the very same thing, and infact I am going to try this solution. I just put something like "not defined" for that spacific features. Is it possible that there is nothing about not mandatory features for some data? Thanks for your reply. $\endgroup$ – Myke Sep 5 '17 at 19:27
  • $\begingroup$ All methods I am aware of would define an additional value for "empty" or "not defined". After all it is a value and has to be treated like one. I'd say give it a try and see if it helps you in your current scenario. Maybe someone can come up with something better, but I am not aware of non-mandatory values. "Not there" is always a kind of feature. $\endgroup$ – Secespitus Sep 5 '17 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.