2
$\begingroup$

I've got very little experience with neural networks and machine learning in general. I have a need to detect anomalies in XML documents. We have thousands of XML documents classified into 22 types. Each type of XML document is similar in structure. How do I go about detecting anomalies in XML structure among document types?

I've done simple outlier detection in time series and simpler structures, however, I'm lost at how to approach this for a tree-like structure of XML.

Some context:

We've got different vendors that send us these XML files. Each with their own method of generating them. We want to be alerted on possible invalid file structures. We cannot use XML comparison algorithms directly since structures can acceptably vary a bit - a few extra nodes, a missing tag, etc. But the structure variances are consistent as well. So these variances should be trained against and not be classified as outliers during execution.

What we really want to catch are totally different structures - since vendors often send incorrect XML files mistakenly.

$\endgroup$
  • $\begingroup$ Have you considered parsing the files and storing them in data structures like arrays, or much better, a data frame? $\endgroup$ – Dawny33 Sep 11 '15 at 13:22
  • $\begingroup$ I'm not sure how I could store a tree-like structure in a data frame $\endgroup$ – gAMBOOKa Sep 11 '15 at 13:27
  • $\begingroup$ It may help us to provide a better answer if you could tell us a little more about the context- is there a particular kind of anomaly that you need to detect? Is there a reason that it would have to be done using a neural network, or is any method fine? $\endgroup$ – Thomas Cleberg Sep 11 '15 at 13:31
  • $\begingroup$ @ThomasCleberg: Any other method is fine as well. I'll edit my question to explain the context a bit $\endgroup$ – gAMBOOKa Sep 11 '15 at 13:32
2
$\begingroup$

A first approach could be to translate your tree structure to a vector, i.e. present every XML file as a vector of features.

Features can be anything (but in the first place, it may be easier to start with numerical features only): the maximum depth of the tree, the count of different attributes of the XML file, the size of the XML file itself, the number of occurrences of specific references in the XML .

Now, your data will look like a big n (number of XML files) p (number of features you extracted) matrix on which you can apply usual outlier detections methods.

$\endgroup$
  • $\begingroup$ I like your approach and I think that can work. As for your edit - by valid, I mean 'classifiable as a predefined type of xml file' - not syntactically valid. $\endgroup$ – gAMBOOKa Sep 11 '15 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.