Detecting anomalies / outliers in XML structure I've got very little experience with neural networks and machine learning in general. I have a need to detect anomalies in XML documents. We have thousands of XML documents classified into 22 types. Each type of XML document is similar in structure. How do I go about detecting anomalies in XML structure among document types?
I've done simple outlier detection in time series and simpler structures, however, I'm lost at how to approach this for a tree-like structure of XML.
Some context:
We've got different vendors that send us these XML files. Each with their own method of generating them. We want to be alerted on possible invalid file structures. We cannot use XML comparison algorithms directly since structures can acceptably vary a bit - a few extra nodes, a missing tag, etc. But the structure variances are consistent as well. So these variances should be trained against and not be classified as outliers during execution.
What we really want to catch are totally different structures - since vendors often send incorrect XML files mistakenly.
 A: A first approach could be to translate your tree structure to a vector, i.e. present every XML file as a vector of features.
Features can be anything (but in the first place, it may be easier to start with numerical features only): the maximum depth of the tree, the count of different attributes of the XML file, the size of the XML file itself, the number of occurrences of specific references in the XML .
Now, your data will look like a big n (number of XML files) p (number of features you extracted) matrix on which you can apply usual outlier detections methods.
A: Did you consider using XML validation, for example, defining an XML Schema XSD (which may include some lax parts), or would it be too much work to specify the minor variations? But then, is it less work to set it up for machine learning? If there's some cooperation established with the suppliers, you may even provide them such a Schema to improve their output.
You could also use some XML library to parse these files, and react to certain elements, dispatch these to a handler, perform a few checks, and proceed. A more "rule"-/event-based approach? Potentially including some counters/toggle-flags to recognize the absence of certain elements/values.
One concern may be to update/maintain/correct/expand these checks, especially if somebody else eventually is expected to do it.
