In a previous question I asked about tools for editing CSV files.
For some applications a dedicated database management system is what is needed. However, for small scale data analysis projects something more light weight seems more suitable.
Consider the following criteria for evaluating a file format:
- reliabile: the data entered should stay true to what has been entered; data should open consistently in different software;
- simple: it would be nice if the file format is easy to understand and ideally be readable with a simple text editor; it should be easy to write a simple program to read and write the format.
- open: the format should be open
- interoperable: the file format should be supported by many systems
I find tab and comma separated value formats fail on the reliability criterion.
Although I suppose I could blame the importing and exporting programs rather than the file format.
I often find myself having to make little adjustments to the options in
read.table in order to prevent some strange character from breaking the loading of the data frame.
- Which file format best meets these needs?
- Is Data Interchange Format a better alternative? or does it have its own problems?
- Is there some other format that is preferable?
- Am I unfairly evaluating TSV and CSV? Is there a simple set of tips for working with such files that make the file format more reliable?