Problem: Lets say there is a staging table with four columns. There are incoming CSV files that arrive daily from thousands of sources. They all have different number of columns and the names of the columns are different for files coming from different sources. I have to write an ML program that identifies the four required columns from these CSVs and recommend a mapping to the target tables.

My Solution(1) : I am trying to use SVM to classify the columns as text classification. Example: I take the four required columns from 5 CSVs and convert them to TermDocumentMatrix, feed them to SVM and train a model to identify which tag these columns could be associated with. The tags being the four column names from the target table.

Road block : The model trains fine. But when I use predict function, it throws the error 'test data does not match model'. And expectedly so, because the test data has other columns too, and other factors, which leads to a different structure of the TermDocumentMatrix

Solution(2) : Use the supervised version Latent Dirichlet Allocation (LDA) package, called sLDA, which is widely used for topic modeling. But due to a lack of understanding of its functioning, I have not been able to use it so far. Especially the parameters 'annotations' and 'variance'.

I am new to ML so any advice would be much appreciated. Also, is this really a problem of text classification?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.