# Lifestyles & Discussion > Science & Technology >  free real-time or streaming data sources?

## tangent4ronpaul

Looking for sources of stream data.  All types.  Few questions:

The last time I looked into API's, they were all pretty limited as to the number of queries you could make in a day.  I've seen recent reference that this has been lifted, some places.  Anyone know what the limitations are at different sites?  Will any of them give you a stream?

Twitter used to have 2 publicly available streams, Google was one of them - think the other was through Yahoo, and they both went away.  Anyone know if there is a free one out there now?  First post says Twitter has an open one.  Unsure on the date.

Anyone know of a good public domain spider out there?  Ditto for scraper?

Is there an open UPI full text site?  I used to know of one, but it went away.

Does anyone know about capturing subtitles for the hearing impaired/deaf off broadcast streams?  Also the "ticker" of latest news on news broadcasts.  The thing has a weird name, and I'm spacing on it right now.  Anyway, suggestions for particular languages/libraries?  I kinda like C, PERL, and Python and dislike javascript and C++.  Have a very rudimentary understanding of asm, unrolling loops and bit operators for fast math, but they are all interesting.  Anyways, pointers appreciated.

This is what I found so far:

★
Where can I find public or free real-time or streaming data sources?
http://www.quora.com/Where-can-I-fin...-data-sources#

Beyond Twitter's streaming API and a few USGS data sources (RSS feeds of earthquake events, for example), what public and/or free data sources are available via the public internet? For example, a feed of the latest Tornado warnings or watches would be a great data set to play with.



There is an hourly updated feed of weather observations and forecasts from the UK Metoffice released as Open Data here: 3 hourly weather forecast and observational data - UK locations
http://data.gov.uk/dataset/metoffice_uklocs3hr_fc

There is also an IRC chan with the live log of wikipedia edits on the #en.wikipedia channel of the  irc.wikimedia.org server. See the following page for more details:  IRC/Channels - Meta
https://meta.wikimedia.org/wiki/IRC/...Recent_changes

Meetup is providing the streaming data of the RSVP's made on their site here: Streaming MEMEs. The APIs for accessing that streaming data is also made available here: Using the Meetup API
http://meetup.github.io/stream/rsvpTicker/
http://www.meetup.com/meetup_api/doc...ps/#websockets

Where can I find sources of high-volume "firehose"-like streaming data?
http://www.quora.com/Real-time-Analy...streaming-data

For example, I am already using real-time stock data from my brokerage. I've connected to the twitter samplehose. Data sources don't have to be 100% public, they can be behind a paywall.

1> Real-Time-Bidding traffic from major exchanges like AdX, AdMeld etc., would be a few billion data-points per day.
2> Astronomy sensor data from telescopes, both terrestrial and out-in-space, would be really large and high dimensional.
3> Video Streams from security-cameras in airports, shopping-malls etc.,

Where can I find large datasets open to the public?
http://www.quora.com/Data/Where-can-...-to-the-public

Just a TON of links!!!!

OK, It's not realtime, but gives you a chance to play with Hapoop and MapReduce, which are related...

Where can I get free real-time flight tracking data via an api?
http://www.quora.com/Where-can-I-get...ata-via-an-api

Here: http://flightaware.com/live/

FlightStats has the broadest range of flight and airport related APIs: flight status and track, schedules/connections, airports, alerts, delay index, weather, ratings etc. Plus FlightStats has global coverage - none of their competitors do
http://developer.flightstats.com/

Does anyone know of other data sources?

-t

----------


## CPUd

I use scrapy for all my scraping these days:

http://scrapy.org/

----------


## tangent4ronpaul

ty CPUd
+rep

anyone else?

-t

----------

