1
$\begingroup$

I am working in a project where I need to extract data from lots of files provided to me in different formats (.exe, .doc, .txt, .dbf, and even .pdf). The data I am interested in are water quality and hydrological variables measured at different sites, with different periodicities (monthly, daily, hourly) and sometimes at different depths. The organization of the files is not easy either. For example, some Excel files contain several tables in the same sheet.

So, I need some software, R package or Python module that can help me organize this mess and extract the relevant data. What do you suggest?

$\endgroup$
  • 1
    $\begingroup$ I somewhat doubt that a single package or even several packages will solve this problem. You'll probably need to combine many tools in a fairly ad hoc manual way. You may want to ask a more specific question since general advice will not be very helpful. $\endgroup$ – dsaxton Sep 22 '16 at 13:02
  • $\begingroup$ If you're looking for a single thing that will extract and prep all your data, you're looking for an assistant that you can assign the job to. You can code your own solution if you know a programming language (and can compile code on your computer), or you can extract the data from each file manually. Which of those is better/easier will depend on the skills you already have, how diversely formatted your tables are, and how many tables you need to process. $\endgroup$ – Upper_Case Sep 22 '16 at 13:50
  • $\begingroup$ @Upper_Case : Well, actually I hoped to avoid that. My ideal software would analyse all the entry files to see which blocks of data had similar structure and held similar kinds of data. Helpful if it extracted tables from doc files and found repeated blocks of data, too. Then I could take over to tell the program how to parse the data from the different types of structure, and let it do the job. With all this big data thing I thought someone would already have done something similar... $\endgroup$ – lodebari Sep 23 '16 at 9:28
  • $\begingroup$ @Iodebari: Things like that have definitely been done, but not necessarily in an implementation that will meet your needs right out of the box and that is available to you. There's no harm in looking for something like that, of course, but the specific data you're looking to extract and the specific formats and layouts of the files may or may not match any existing, publically available solution. $\endgroup$ – Upper_Case Sep 23 '16 at 12:50
1
$\begingroup$

I would use MySQL to organize the data and then access it through R or Python (see RMySQL package for R or MySQLdb for Python). MySQL allows you to structure and manage your complex data, and for the actual statistical analysis you can fetch only the data you need.

To complete the answer (thanks to @Ian_Fin for the observation): For data extraction I think you need to create a php/java program to parse your different files and then save them to the database.

$\endgroup$
  • $\begingroup$ @Ian_Fin: I doubt there is a solution in, say R, for the variety of formats lodebari indicates. My solution was related to the way s/he "can organize the mess". For data extraction probably s/he needs to create a php / java program to parse them and then save them to the database. $\endgroup$ – Luminita Sep 22 '16 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.