3
$\begingroup$

I've got very little experience with neural networks and machine learning in general. I have a need to detect anomalies in XML documents. We have thousands of XML documents classified into 22 types. Each type of XML document is similar in structure. How do I go about detecting anomalies in XML structure among document types?

I've done simple outlier detection in time series and simpler structures, however, I'm lost at how to approach this for a tree-like structure of XML.

Some context:

We've got different vendors that send us these XML files. Each with their own method of generating them. We want to be alerted on possible invalid file structures. We cannot use XML comparison algorithms directly since structures can acceptably vary a bit - a few extra nodes, a missing tag, etc. But the structure variances are consistent as well. So these variances should be trained against and not be classified as outliers during execution.

What we really want to catch are totally different structures - since vendors often send incorrect XML files mistakenly.

$\endgroup$
4
  • $\begingroup$ Have you considered parsing the files and storing them in data structures like arrays, or much better, a data frame? $\endgroup$
    – Dawny33
    Commented Sep 11, 2015 at 13:22
  • $\begingroup$ I'm not sure how I could store a tree-like structure in a data frame $\endgroup$
    – gAMBOOKa
    Commented Sep 11, 2015 at 13:27
  • $\begingroup$ It may help us to provide a better answer if you could tell us a little more about the context- is there a particular kind of anomaly that you need to detect? Is there a reason that it would have to be done using a neural network, or is any method fine? $\endgroup$ Commented Sep 11, 2015 at 13:31
  • $\begingroup$ @ThomasCleberg: Any other method is fine as well. I'll edit my question to explain the context a bit $\endgroup$
    – gAMBOOKa
    Commented Sep 11, 2015 at 13:32

2 Answers 2

3
$\begingroup$

A first approach could be to translate your tree structure to a vector, i.e. present every XML file as a vector of features.

Features can be anything (but in the first place, it may be easier to start with numerical features only): the maximum depth of the tree, the count of different attributes of the XML file, the size of the XML file itself, the number of occurrences of specific references in the XML .

Now, your data will look like a big n (number of XML files) p (number of features you extracted) matrix on which you can apply usual outlier detections methods.

$\endgroup$
1
  • $\begingroup$ I like your approach and I think that can work. As for your edit - by valid, I mean 'classifiable as a predefined type of xml file' - not syntactically valid. $\endgroup$
    – gAMBOOKa
    Commented Sep 11, 2015 at 15:51
1
$\begingroup$

Did you consider using XML validation, for example, defining an XML Schema XSD (which may include some lax parts), or would it be too much work to specify the minor variations? But then, is it less work to set it up for machine learning? If there's some cooperation established with the suppliers, you may even provide them such a Schema to improve their output.

You could also use some XML library to parse these files, and react to certain elements, dispatch these to a handler, perform a few checks, and proceed. A more "rule"-/event-based approach? Potentially including some counters/toggle-flags to recognize the absence of certain elements/values.

One concern may be to update/maintain/correct/expand these checks, especially if somebody else eventually is expected to do it.

$\endgroup$
1
  • $\begingroup$ +1 it's odd that nobody posted this answer for such a long time. This problem doesn't seem to need machine learning. $\endgroup$
    – Tim
    Commented Dec 16, 2022 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.