1
$\begingroup$

I am working in a project where I need to extract data from lots of files provided to me in different formats (.exe, .doc, .txt, .dbf, and even .pdf). The data I am interested in are water quality and hydrological variables measured at different sites, with different periodicities (monthly, daily, hourly) and sometimes at different depths. The organization of the files is not easy either. For example, some Excel files contain several tables in the same sheet.

So, I need some software, R package or Python module that can help me organize this mess and extract the relevant data. What do you suggest?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I somewhat doubt that a single package or even several packages will solve this problem. You'll probably need to combine many tools in a fairly ad hoc manual way. You may want to ask a more specific question since general advice will not be very helpful. $\endgroup$
    – dsaxton
    Commented Sep 22, 2016 at 13:02
  • $\begingroup$ If you're looking for a single thing that will extract and prep all your data, you're looking for an assistant that you can assign the job to. You can code your own solution if you know a programming language (and can compile code on your computer), or you can extract the data from each file manually. Which of those is better/easier will depend on the skills you already have, how diversely formatted your tables are, and how many tables you need to process. $\endgroup$
    – Upper_Case
    Commented Sep 22, 2016 at 13:50
  • $\begingroup$ @Upper_Case : Well, actually I hoped to avoid that. My ideal software would analyse all the entry files to see which blocks of data had similar structure and held similar kinds of data. Helpful if it extracted tables from doc files and found repeated blocks of data, too. Then I could take over to tell the program how to parse the data from the different types of structure, and let it do the job. With all this big data thing I thought someone would already have done something similar... $\endgroup$
    – lodebari
    Commented Sep 23, 2016 at 9:28
  • $\begingroup$ @Iodebari: Things like that have definitely been done, but not necessarily in an implementation that will meet your needs right out of the box and that is available to you. There's no harm in looking for something like that, of course, but the specific data you're looking to extract and the specific formats and layouts of the files may or may not match any existing, publically available solution. $\endgroup$
    – Upper_Case
    Commented Sep 23, 2016 at 12:50

1 Answer 1

Reset to default
1
$\begingroup$

I would use MySQL to organize the data and then access it through R or Python (see RMySQL package for R or MySQLdb for Python). MySQL allows you to structure and manage your complex data, and for the actual statistical analysis you can fetch only the data you need.

To complete the answer (thanks to @Ian_Fin for the observation): For data extraction I think you need to create a php/java program to parse your different files and then save them to the database.

Improve this answer
$\endgroup$
1
  • $\begingroup$ @Ian_Fin: I doubt there is a solution in, say R, for the variety of formats lodebari indicates. My solution was related to the way s/he "can organize the mess". For data extraction probably s/he needs to create a php / java program to parse them and then save them to the database. $\endgroup$
    – NRLP
    Commented Sep 22, 2016 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.