Return to Question

In a previous question I asked about tools for editing CSV files.

Gavin linked to a comment on R Help by Duncan Murdoch suggesting that Data Interchange Format is a more reliable way to store data than CSV.

For some applications a dedicated database management system is what is needed. However, for small scale data analysis projects something more light weight seems more suitable.

Consider the following criteria for evaluating a file format:

• reliabile: the data entered should stay true to what has been entered; data should open consistently in different software;
• simple: it would be nice if the file format is easy to understand and ideally be readable with a simple text editor; it should be easy to write a simple program to read and write the format.
• open: the format should be open
• interoperable: the file format should be supported by many systems

I find tab and comma separated value formats fail on the reliability criterion. Although I suppose I could blame the importing and exporting programs rather than the file format. I often find myself having to make little adjustments to the options in read.table in order to prevent some strange character from breaking the loading of the data frame.

Questions

• Which file format best meets these needs?
• Is Data Interchange Format a better alternative? or does it have its own problems?
• Is there some other format that is preferable?
• Am I unfairly evaluating TSV and CSV? Is there a simple set of tips for working with such files that make the file format more reliable?

In a previous question I asked about tools for editing CSV files.

Gavin linked to a comment on R Help by Duncan Murdoch suggesting that Data Interchange Format is a more reliable way to store data than CSV.

For some applications a dedicated database management system is what is needed. However, for small scale data analysis projects something more light weight seems more suitable.

Consider the following criteria for evaluating a file format:

• reliabile: the data entered should stay true to what has been entered; data should open consistently in different software;
• simple: it would be nice if the file format is easy to understand and ideally be readable with a simple text editor; it should be easy to write a simple program to read and write the format.
• open: the format should be open
• interoperable: the file format should be supported by many systems

I find tab and comma separated value formats fail on the reliability criterion. Although I suppose I could blame the importing and exporting programs rather than the file format. I often find myself having to make little adjustments to the options in read.table in order to prevent some strange character from breaking the loading of the data frame.

Questions

• Which file format best meets these needs?
• Is Data Interchange Format a better alternative? or does it have its own problems?
• Is there some other format that is preferable?
• Am I unfairly evaluating TSV and CSV? Is there a simple set of tips for working with such files that make the file format more reliable?

In a previous question I asked about tools for editing CSV files.

Gavin linked to a comment on R Help by Duncan Murdoch suggesting that Data Interchange Format is a more reliable way to store data than CSV.

For some applications a dedicated database management system is what is needed. However, for small scale data analysis projects something more light weight seems more suitable.

Consider the following criteria for evaluating a file format:

• reliabile: the data entered should stay true to what has been entered; data should open consistently in different software;
• simple: it would be nice if the file format is easy to understand and ideally be readable with a simple text editor; it should be easy to write a simple program to read and write the format.
• open: the format should be open
• interoperable: the file format should be supported by many systems

I find tab and comma separated value formats fail on the reliability criterion. Although I suppose I could blame the importing and exporting programs rather than the file format. I often find myself having to make little adjustments to the options in read.table in order to prevent some strange character from breaking the loading of the data frame.

Questions

• Which file format best meets these needs?
• Is Data Interchange Format a better alternative? or does it have its own problems?
• Is there some other format that is preferable?

In a previous question I asked about tools for editing CSV files.

Gavin linked to a comment on R Help by Duncan Murdoch suggesting that Data Interchange Format is a more reliable way to store data than CSV.

For some applications a dedicated database management system is what is needed. However, for small scale data analysis projects something more light weight seems more suitable.

Consider the following criteria for evaluating a file format:

• reliabile: the data entered should stay true to what has been entered; data should open consistently in different software;
• simple: it would be nice if the file format is easy to understand and ideally be readable with a simple text editor; it should be easy to write a simple program to read and write the format.
• open: the format should be open
• interoperable: the file format should be supported by many systems

I find tab and comma separated value formats fail on the reliability criterion. Although I suppose I could blame the importing and exporting programs rather than the file format. I often find myself having to make little adjustments to the options in read.table in order to prevent some strange character from breaking the loading of the data frame.

Questions

• Which file format best meets these needs?
• Is Data Interchange Format a better alternative? or does it have its own problems?
• Is there some other format that is preferable?
• Am I unfairly evaluating TSV and CSV? Is there a simple set of tips for working with such files that make the file format more reliable?