Locating freely available data samples

Question

I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics. While the method works well enough with artificial data samples (i.e. datasets created specifically for the purpose of identifying and segregating subsets of the population), I'd like to try testing it with live data.

What I'm looking for is a freely available (i.e. non-confidential, non-proprietary) data source. Preferably one containing bimodal or multimodal distributions or being obviously comprised of multiple subsets that cannot be easily pulled apart via traditional means. Where would I go to find such information?

Answer 1

Also see the UCI machine learning Data Repository.

http://archive.ics.uci.edu/ml/

Answer 2

The following list contains many data sets you may be interested:

• America's Best Colleges - U.S. News & World Reports
• American FactFinder
• The Baseball Archive
• The Bureau of Justice Statistics
• The Bureau of Labor Statistics
• The Bureau of Transportation Statistics
• The Census Bureau
• Data and Story Library (DASL)
• Data Sets, UCLA Statistics Department
• DIG Stats
• Economic Research Service, US Department of Agriculture
• Energy Information Administration
• Eurostat
• Exploring Data
• FedStats
• The Gallop Organization
• International Fuel Prices
• Journal of Statistics Education Data Archive
• Kentucky Derby Race Statistics
• National Center for Education Statistics
• National Center for Health Statistics
• National Climatic Data Center
• National Geophysical Data Center
• National Oceanic and Atmospheric Administration
• Sports Data Resources
• Statistics Canada
• StatLib---Datasets Archive
• UK Government Statistical Service
• United Nations: Cyber SchoolBus Resources

Answer 3

See my response to "Datasets for Running Statistical Analysis on" in reference to datasets in R.

Answer 4

Gapminder has a number (430 at the last look) of datasets, which may or may not be of use to you.

Answer 5

World Bank offers quite a lot of interesting data and has been recently very active in developing nice API for it.

Also, commugrate project has an interesting list available.

For US health related data head for Health Indicators Warehouse.

Daniel Lemire's blog points to few interesting examples (mostly tailored towards DB research) including Canadian Census 1880 and synoptic cloud reports.

And as for today (03/04/2012) US 1940 census records are also available to download.

Answer 6

MLComp has quite a few interesting datasets, and as a bonus your algorithm will get ranked if you upload it.

Answer 7

Start R and type data(). This will show all datasets in the search path. Many additional datasets are available in add-on packages. For example, there are some interesting real-world social science datasets in the AER package.

Answer 8

http://lib.stat.cmu.edu/DASL/

Answer 9

NIST provides a Reference Dataset archive.

Answer 10

http://www.reddit.com/r/datasets and also, http://www.reddit.com/r/opendata both contain a constantly growing list of pointers to various datasets.

Answer 11

http://mathforum.org/workshops/sum96/data.collections/datalibrary/data.set6.html

Answer 12

Timetric provides a web interface to data and provide a list of the publicly available data sets they use

Answer 13

Here's another list.

Answer 14

Adding a couple to the list:

• Lots of in-depth financial data on publicly-traded companies, going back many decades: http://www.mergent.com/servius

• Rich information on 16+ million businesses in the US: http://compass.webservius.com

Both available via a REST API and have free trial plans.

Answer 15

• http://www.freebase.com/

Answer 16

This is probably the most complete list you'll find: Some Datasets Available on the Web

Answer 17

Peter Skomoroch maintains a list of datasets at http://www.datawrangling.com/some-datasets-available-on-the-web. Many of the links provided as to places that list datasets.

Answer 18

Usage Over Time

A very large Excel spreadsheet available for download containing data points for all online activities, with user demographics, over time. Please read Tip Sheet (below) before downloading or using this spreadsheet.

http://pewinternet.org/Trend-Data/Usage-Over-Time.aspx

Answer 19

http://www.ckan.net has a number of datasets too.

http://www.biotorrents.net/browse.php is also starting to have quite a large amount of BIG datasets.

Answer 20

Data sets from seminal book A handbook of small data sets are available here.