3
$\begingroup$

I'm looking for common standards (file formats) for how most statistics software packages import data.

The reason is that I'm going to make an export function for a simulation software package.

The data to be exported consists of time-series of (1 or more) values for all members of a population. The population size can be in the 100,000's. The length of the time-series is typically in the 1,000's. The number of values per member per timestep can range from 1 to 20.

How is such data typically imported by statistics packages?

$\endgroup$
  • 3
    $\begingroup$ The master rule is not XLS. $\endgroup$ – user88 Feb 10 '12 at 10:15
  • 2
    $\begingroup$ Watch out for date handling, it's easy to run into problems with that. $\endgroup$ – Michelle Feb 10 '12 at 11:17
  • $\begingroup$ csv is the safest bet for compatibility. Different software may have formats with particular optimisations/features, though the cynic may argue that its more to do with creating a dependency on that software... $\endgroup$ – James Feb 10 '12 at 13:34
8
$\begingroup$

To expand on @King's answer a bit:

A convenient 'lowest common-denominator' format is something like:

  • Comma-separated values (CSV)
  • Variable names in first 'header' line
  • variable names alphanumeric starting with a letter, not case dependent, no spaces or underscores or dots, 16 characters max
  • same number of comma-separated fields on each line
  • 'Long' format, i.e. multiple lines per subject, one line for each timepoint within each subject

e.g.:

 personid,time,vara,varb
 1001,1,4.322,6
 1001,2,5.645,7
 1001,3,6.332,10
 1003,1,8.434,2
 1003,2,5.342,4
 ...
| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ (+1) I would add that in case of factors, labels should be encoded directly in the csv file, when possible; otherwise, rely on a separate codebook. (This might also be used to provide extended labels, i.e. more explicit description of the variable than a 16 char. long descriptor). Likewise, units of measurement for a given variable should be constant across records (when possible), and documented in a separate file. If providing the user with a single file is mandatory, we can use lines before or after the header line to give all those information; this might be skipped when reading the file. $\endgroup$ – chl Feb 10 '12 at 12:39
  • $\begingroup$ excellent -this is exactly the information I needed $\endgroup$ – nominator Feb 10 '12 at 13:47
3
$\begingroup$

I'm working in a department where different people use different softwares (R, Stata, SAS, SPSS, etc); we are quite happy with passing data around using CSVs.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

CSV is a time tested format. But avoid a world of headaches and Use the IETF RFC 4180 standard for CSV files, particularly if data comes from external sources where you don't control the string formats on input. i.e.

  • if the value has a comma or line break, surround it with double quotes

  • If it has a double quote, put a double quote before it to escape the double quote. etc.

(Or globally strip out linebreaks, commmas and quotes).

That said, CSV loses critical information, and you should consider SPSS (.sav or .por) format or XML W3C standard for relational tables. Both include meta data that CSV loses, like variable types (i.e. zip codes are actually strings with leading zeros, but look like integers, and the leading zeros get lost in input downstream). SPSS data files also include variable and value labels.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.