I am working on a data-visualization web application that looks at trends in American news media coverage over time. It takes a keyword (or keywords), a date range, and a time increment as parameters and generates a line graph and data set of how often that keyword appears in news articles during the given time period (with one data point for each time increment- either a month or year). So if you search "Mitt Romney" in 2012, you'll see a line that gradually fluctuates during primary season and peaks around the November election, before falling off, and a table with each data point's value and date range.
Right now I am manually scraping this information (no API) from an external web site. However this is inefficient, slow, and the site's ToS are fairly restrictive. I would ideally like to swap out this backend with a new data source, but I am having trouble finding sources that fit my needs.
I need a data source that:
- Shows the number of results of a search. This can be subtle or a non-central function. Search engines or APIs that show things like "Results 1-10 of 1242" fulfill this criteria
- Supports specific date ranges. So I can get results from "April 1 2013 to April 1 2014" rather than just "last 30 days" or something
- Is cheap or free, and has non-restricted ToS.
This has proven fairly complicated. I've looked at the following and found:
- Google News API is deprecated.
- Bing News API seems like it would fit this functionality reasonably well, but they have a low per-second query limit, their free monthly limit is low, and their prices to get higher limits are oppressively high.
- Faroo's API is the closest to usable that I've found so far. They support news search by keyword, are free, and have a high query limit. The number of articles that are returned by a query is capped at 10, but it shows the total number of results for the search criteria, which is exactly what I'm looking for. However Faroo does not allow you to search by date; articles returned store their publication dates, but since only 10 articles are returned, this does not allow for effective use of frequency over time data, which is critical.
- I've investigated a number of other smaller search engines, but they all lack critical features.
This is a fairly stringent set of requirements and I've therefor had trouble finding an actual API that fits the bill. Is anyone aware of APIs or databases that I can use that support the above functionality? Ideally I need a free way of pairing keyword, date range and article frequency, but haven't found any broadly aggregating data source that I can use.
Any suggestions greatly appreciated.