2
$\begingroup$

I would like to match two schemas coming from different sources (DB). For this I would like to try some machine learning algorithm to come up with a process to decided which column in a table could correspond to the same key. For example it's very likely that FNameand FirstName should be a match and correspond both to the same "real world key" First Name. I would therefore do the following:

  • For each column (key) extract certain features. Some of them may be based only on the key itself, others may use actual values of that key to generate certain features.
  • I thought I will define features based on their data type, i.e. have certain feature extraction for strings, for numeric etc.
  • Once I've generated all the feature for each column I can train a classifier.
  • Once I have a trained model I can use it to predict for other schema if there is a match

Does the above algorithm make sense? There is one point which confuses me. I'm only interested in matching certain schema like core personal data. To training such a classifier I do need to have some test data. Is the test data in this case different schemas? That would mean I need many schemas about personal data which could be very difficult to find. Or how else could I train the model with just a few (say less than 10) schemas at hand?

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

My initial thoughts

This problem seems easier to be solved by actually looking and understanding the schema from the DB's that you are using. ML is usually a good idea where the cost of a mistake ( false positive or false negative ) isn't too high.

In your case, a false negative (predicting two columns are not a match when they are) implies on not being able to run you application at all with those two features; a false positive (predicting two columns are a match when they aren't) means that the application will run with faulty data.

Both of those seem a high cost to run an analysis/application based on ML.

If I had to do it

I would consider simple matching features before diving into a machine learning algorithm for 2 reasons:

  • you don't seem to have enough training data
  • you don't have a benchmark for comparing performance

So I would go for setting up a benchmark with simple heuristics so that you can set up you analysis/application, and improve it with ML as needed.

The simple heuristics I suggest are:

  • similar descriptive statistics for numerical values
  • most/least common for categorical values
  • string similarity on the column names

Etc...

With that, you can start to have a comparison case before running a ML algorithm, which it self can have many problems associated with it, that are not related with the application you might be aiming.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. This is exactly how I was thinking about it. The simple heuristic you've mentioned would be the features I've created. But I need then to run a classifier to come up with a decision ln these features or am I wrong $\endgroup$
    – math
    Jul 18, 2017 at 16:22
  • $\begingroup$ If you don't have any clue about how to come up with simple heuristics rules, then yes, you'd have to run a classifier (like a simple decision tree). However I find it hard to believe that you have no clue on how the match features ('like title similarity has to be above 80%' or 'the mean should be at least 1.5 apart'). Will these simple heuristics be bad? Yes, but it will kick of things and set a benchmark. $\endgroup$ Jul 18, 2017 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.