2
$\begingroup$

Complete newbie here. I have a node script that parses thousands of large html docs and extracts data. My algorithm isn't perfect (the html docs are not similarly structured). I've been reading about supervised learning and other machine learning techniques and was hoping for some guidance.

Here's an example input/output:

var vector = readFile('file.html');
var desiredOutput = { 
    'Net Income from herding cats': [
        { 2015, '79268' }, 
        { 2014, '79268' }, 
        { 2013, '79268' }
    ]
};
var trainedModel = trainInput(vector, desiredOutput);

Example input for file.html

<table>
    <tbody>
        <tr>
            <td>
                <div><font>&nbsp;</font></div>
            </td>
            <td colspan="3">
                <div><font>2015</font></div>
            </td>
            <td>
                <div><font>&nbsp;</font></div>
            </td>
            <td colspan="3">
                <div><font>2014</font></div>
            </td>
            <td>
                <div><font>&nbsp;</font></div>
            </td>
            <td colspan="3">
                <div><font>2013</font></div>
            </td>
        </tr>
        <tr>
            <td>
                <div><font>Net Income from herding cats</font></div>
            </td>
            <td>
                <div><font>$</font></div>
            </td>
            <td>
                <div><font>79,268</font></div>
            </td>
            <td>
                <div><font><br></font></div>
            </td>
            <td>
                <div><font>&nbsp;</font></div>
            </td>
            <td>
                <div><font>$</font></div>
            </td>
            <td>
                <div><font>70,080</font></div>
            </td>
            <td>
                <div><font><br></font></div>
            </td>
            <td>
                <div><font>&nbsp;</font></div>
            </td>
            <td>
                <div><font>$</font></div>
            </td>
            <td>
                <div><font>60,903</font></div>
            </td>
            <td>
                <div><font><br></font></div>
            </td>
        </tr>
    </tbody>
</table>

This table for is simplified for demonstration purposes. All of these documents are terribly structured and can have 100s of tables. Each table has different formatting. There are no reliable classes or ids to rely on so using jquery/cheerio is not a perfect solution. I've been using a dictionary search to select the appropriate tables and then some regex to extract keys/values. My accuracy is fairly good, but I'd like to make it better if possible.

My questions are:

  1. What ML technique is applicable for inferring a function based on a set of examples?
  2. Is there a preferred library for this?*

[Note*] It is not required to be fast as I am not doing realtime analysis. I do not need a UI or data viz. I'm using node but can really use anything if there are better options in python/java.

$\endgroup$
  • 1
    $\begingroup$ "My accuracy is fairly good": Accuracy of what? What is the goal of your algorithm? Classification? Clustering? Prediction? In general, there is no restriction for doing machine learning in Javascript, but there are not many libraries available and you will need to re-invent the wheel many many times. Python for instance provides many very advanced libraries for machine learning and can be worth learning. Sklearn is one of the general Python machine learning libraries. There are others but it all depends on what you are trying to achieve, which is not clear to me. $\endgroup$ – Eskapp Nov 23 '16 at 16:54
  • 1
    $\begingroup$ The accuracy of extracting data sets from these documents. So, let's say every document has many tables and I want certain rows (keys and values) parsed out. I know the keys to look for. I want to create a function such that I can feed it more of these documents and have it extract the key/values that I need. See desiredOutput above. $\endgroup$ – Lasso138 Nov 23 '16 at 19:10
  • $\begingroup$ I think my assumptions about ML may be wrong. I think what I have been looking for was a good way to clean a lot of data -- which comes from mostly inconsistent and dirty sources (dissimilarly structured documents). I'm not sure though. $\endgroup$ – Lasso138 Nov 23 '16 at 20:41
  • $\begingroup$ I think so too. Maybe you could edit your question with respect to this. $\endgroup$ – Eskapp Nov 23 '16 at 22:19
  • $\begingroup$ Here is a company which are doing it seems you want: diffbot.com and here is a duplicate Q on SO which have 7 answers: stackoverflow.com/questions/13336576/… $\endgroup$ – kjetil b halvorsen Feb 13 '18 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.