For far too long now RSS has been used in ways that don't really tap its true potential. Being able to syndicate my favorite headlines or blog posts is great. In fact, it helped to kick off a revolution in personal on-line publishing that is still growing and evolving. But I want so much more.
It's not for lack of vision. Back in 2005, Adam Bosworth painted a vision that eventually manifested itself as GData. I wrote about that a bit in Google's GData, MySQL, and the Future of on-line Databases.
It's not for lack of data either. You can get RSS output from lots of non-news and non-blog stuff. Everything from classifieds on eBay and craigslist to Bugzilla, Wikis, and so on.
The problem has been a lack of good tools for pulling it all together. In the Unix world, we often connect sources of data to filters and utilities using pipes. A pipe is a way of constructing ad-hoc workflows composed of any number of inputs, filters, and manipulation tools. And the beauty of the whole system is that they all use a very simple input and output method, so there's a nearly infinite set of ways you can combine and recombine them.
On the web, however, it's harder. There are data sources and feeds, but until now we've had no pipes! Pulling together and integrating data sources using JavaScript isn't on the client for the faint of heart. The browser isn't the same as a Unix command-line, so building mashups has been more frustrating and time consuming that it needs to be--especially for Unix people like me.
Yahoo! Pipes is one of a very small set of completely amazing on-line data manipulation and data mashup environments that can really change the way we work with on-line data sources. (The others are DabbleDB and Dapper.)
Yahoo! Pipes is...
Pipes is a hosted service that lets you remix feeds and create new data mashups in a visual programming environment. The name of the service pays tribute to Unix pipes, which let programmers do astonishingly clever things by making it easy to chain simple utilities together on the command line.
So if you're interested in an interactive on-line data mashup construction set, check it out and have a look at the existing pipes developers have created. I think it's one of the coolest web apps we've released in a long time, and I'm not just saying that because I know the guys who built it. Not only does it make hard things easy (like Perl has done for years), the user interface kicks ass too. It's exciting both from a technological point of view and because of the implications for the web as a whole. :-)
For another great take on this, check out what Tim O'Reilly has to say about Pipes. I don't think he's over-selling it by calling Pipes "milestone in the history of the internet." It's seriously cool stuff.
I'm hoping we can pull together some demos and interviews with the team next week. Watch the Yahoo! Developer Network blog for more...
See Also:
- Remixing the Web with Yahoo! Pipes by Ed Ho, one of the amazing Pipes developers
- Pipes! by Bradley Horowitz
- Yahoo Pipes - The Internet is a Series of Them by Pete Cashmore
- Yahoo Pipes by Glenn Slaven
- Review: Yahoo! Pipes by Matt Cutts of Google
- Yahoo Pipes on Gadgetopia
- Pipes: Remixing the Web by Jonathon Trevor, another of the kick-ass Pipes developers
- Yahoo! Pipes remixes the syndicated web by Niall Kennedy
- Yahoo Pipes by Anil Dash
- cat rss | ... by Ryan Kennedy of Y! Mail API fame
- yahoo pipes…rss splicing and more… by Don Loeb of Feedburner
- Yahoo! Pipes by Avi Bryant of DabbleDB
- Yahoo Launches Pipes, an RSS Remixer by Richard MacManus
- Yahoo! Launches Pipes by Nik on TechCrunch
- Yahoo! Pipes: “a milestone in the history of the internet” by Chad Dickerson of the Yahoo! Developer Network
- Pipes! by Pasha Sadri, the man behind Pipes
- Yahoo Pipes becomes belle of the RSS ball by Larry Dignan of ZDNet
- The WOW starts now: Yahoo! Pipes by Garvin Hicking
- Pipes on the Internet: Yahoo Pipes by Ajit K
- Online Visual Programming with Yahoo! Pipes by Heritage Tech
Update @7:20am on Feb 8th: Pipes is currently down. There's a lot of interest, so things are getting tweaked on the backend. I'll post more when we have an update.
Update @7:00am on Feb 9th: Pipes is alive and tuned! Try it out.
Posted by jzawodn at February 07, 2007 08:06 PM
Seems like a really cool idea! I tried it and got "invalid module type: pipe:GipCBi.32xGsFE3JXqIxGw at /home/y/lib/perl5/site_perl/Yahoo/RSS/Pipe.pm line 369." I was trying to hook a yahoo search up to the pipe output.
Very interesting. Today I unsubscribed to IMDB's daily news feed because I'm only interested in the movie review summaries they do every Friday, and it wasn't worth it for me anymore. But, it took me about 10 minutes to develop a pipe that returns an RSS feed of just those reviews. It'll break if they change the way they name their articles, but it's pretty damn impressive nonetheless.
Thanks, Matt. I'll make sure they see the bug.
This stuff is, of course, beta right now. :-)
Congrats, Jeremy!
I think you guys have a winner here.
Best,
-DeWitt
Wow. Excellent App. Endless possibilities .. I love the UI
Thanks Matt!
I believe we've got that bug fixed now, please give it another try.
Very cool service, but I'm getting an error every time I hit the site now:
"Sorry, Unable to process request at this time -- error 999."
I'm guessing this is just due to server load?
Nope, that's not related to server load (usually). I'll shoot you an email to see if we can figure it out.
Looks much better, Edward Ho, thanks! Can you say a word about when pipes are public/private? Maybe saving a search = get an obfuscated url, and publishing it makes it publicly viewable?
Also, do you have to be Yahoo-ID-logged-in to run a pipe, or do you just need to load the obfuscated url?
Matt:
Sure, so pipes are all public in the sense that anyone with the obfuscated url can run and subscribe to it. We did this to facilitate sharing via IM/Email/etc...
You do not need to be logged into Yahoo to do any of the above. You do need to log in if you want to clone a version to modify for yourself.
The "Publish" feature is to make your pipe visible to others via Search and Browse.
the editing ui is very cutting edge, pretty much the limit of what you can stuff into a browser. maybe a little over the limit, it killed firefox2 on freebsd62 for me. but thats fine, if you are going to experiment, go to the frontier or don't bother.
not sure how many people will use and rely on the actual pipes, though that also likely does not matter.
kudos to the team for trying something beyond the triviality of most "mash" initiatives.
Using "RSS" as a generic term for feeds and saying that Perl makes "hard things easy"...?
Yup.
RSS == Feeds, whether they're RSS x.y, Atom, etc. (at least in my head)
And Perl did make hard things easy for a whole lot of the first generation of the Web.
This is amazing! Needs some work, ex. problems with unicode characters, but the whole implementation is great.
Finaly my dream (http://vrypan.net/log/2005/10/31/rip-mix-burn-for-feeds/) comes true...
Now, could you please buy FeedBurner and intergrate their services into Pipes, please?
Heh. You pony up a few hundred million dollars, and I'll see what I can do. :-)
Gald you like it, though.
"Heh. You pony up a few hundred million dollars, and I'll see what I can do. :-)"
Sounds like you guys and FB have been talking :)
That's great. I just played around with and I love it.
But (of course there is a but) it would be great if there would be a sitescraper source could be added to extract data from websites without feeds. You probably didn't do it for IP reasons, I guess.
What I couldn't find is a way to manipulate items of a feed but maybe I haven't looked hard enough. Eg. I mixed several feeds into one and wanted to add the source of an Item to the title...
Life is a pipe...
a proud Yahoo! employee
Pipes is really cool. And what's more its UI is fun and useful at the same time. An as you go preview would be very helpful.
Next stage is to populate the sources options with Dapps (i.e. with all of the web) :)
Kodus to Yahoo on bringing a real innovation
Wow. This is the *perfect* tool for me to illustrate to my coworkers a number of concepts relating to blogging, rss, and the universal database of participation all of us have been building.
Sure you could all of what Pipes provided with some Python or Perl fairly quickly - if you were a scripter like myself.
But this opens it up to almost anyone. And that makes it huge.
On a personal note, this is precisely the kind of tool that will enable me to provide useful views into the conversation talking place in the Philly Future regional blog community.
I don't see an OPML input there, but I've just dipped my toes in this morning.
Great work Yahoo! :)
Pretty original, I love it ! I posted a quick review on AJAX magazine too.
Good to see Yahoo innovating.
No doubt that we are at the beginning of a new area http://brunovernay.wordpress.com/2007/02/08/yahoo-pipes/
At least, that kind of tool may drive the adoption of more "processing friendly" techno, like XHTML and other Semantic-web stuff that only geek seems to care about now.
Sounds like a fabulous service... aside from the fact it's totally dead at the moment. :)
I second Dom. I've been able to actually access 'Pipes' maybe 1 in 4 instances. The only time I was able to actually get in, I got a 'System Error' notice.
Finally, real competition! With FeedDigest we've been doing the generation before this for almost two years now, and now it's time to kick things up a notch or two, which Pipes appears to do. Of course, competition brings the big guns out, so it'll be interesting to see what happens next ;-)
Using RSS, that's a start. How about allowing us to define data types || sources or take a Ning-like approach and let us clone existing sources and modify them?
The first new data type I could think of would be JSON.
Yahoo! This is brilliant. Pipes stands squarely in the middle of where it's all going.
This is what I'm seeing on the pipes pages right now: "Our Pipes are clogged! We've called the plumbers!"
Popularity is a high-class problem! Congrats!
Isn't Pipes sort of like Excel for internet data feeds?
Say, this is "Pipes" thing is same product as Vignette's BIS. The main difference is BIS uses a Windows client and Pipes is web-based.
http://www.vignette.com/dafiles/images/IM0601_ss_BISLg.gif
The quick GAP analysis:
1) Pipes should let a user interrogate a WSDL file. Pipes should then present to the user the fields defined in the WSDL required for input and let the user map the resulting outputs.
2) Vignette BIS integrates with a product called Vignette Builder:
http://www.vignette.com/us/Products/Portals/Vignette+Builder
http://www.vignette.com/dafiles/images/IM0601_ss_BuilderLg.gif
Builder takes the Pipe (err ... BIS output) defined by BIS and lets the user create forms and tables based on the inputs / outputs.
Conclusion: Pipes should let users create rich Portlets (err...WIDGETS) based on their Pipes and publish them to My Yahoo or other blog. Oh wait, it already does!
3) JDBC: BIS lets users utilize a JDBC data source. Easy to add to Pipes.
Not so revolutionary fan boi's, but I'll say that Vignette is screwed and wasted a good piece of tech.
Nice, the pipes are clogged. Doesn't bode well
when Yahoo cannot even maintain uptime with
this.
Just wanted to link to another ITconversation download: Part II of Robert Lefkowitz's Semasiology of Open Source. I highly recommend the whole 3-part series, but part 2 is facsinatingly relevant to what Tim O'Reilly is talking about when he says "mashup programming to the non-programmer", and also some of the other takes on Pipes I've read around the blogosphere today.
Yeah, clogged for me too.
Makes me wonder: why can Google and Yahoo! not prepare well enough for such user demand? Both companies know the demand exists, they should both be well equipped for the service.
Happened with Google Analytics, Gmail, name it you got it...now Pipes too? :(
And I was just about to turn off my computer and go to the gym .. good grief, will the fun toys to play with never stop?
Exciting and promising stuff. I hope Yahoo is going to clarify whether this tool is intended to target only end (power) users or if it's going to be supported in a way that makes it "serious enough" to be used by companies too (think, say, Bloglines vs. Amazon S3).
I can't figure it out :-(
I just want to be able to re-combine a bunch of RSS feeds and create into one like I can with FeedBlendr which unfortunately does weird things with apostrophes.. Can somebody please make a tutorial or show the steps?
Thanks,
-Bob
Are VoIP Bloggers original ?
A fun take using Yahoo Pipes here:
http://corporaterat.blogspot.com/2007/02/piping-hot-are-voip-bloggers-really.html
Jeremy, Yahoo Pipes folks:
Great idea. Very good implementation. Like any IDE, shouldn't there be some way to get help? I was trying to build a pipe but couldn't figure out some details - a help function would have been helpful.
-Anupam
It looks like functionality right now is a little broken as no pipes seem to output much right now.
Besides that... this is some great stuff. Who would have thought... pipe and filter. Now why didn't I think of that!
Great job Yahoo!
Trying to build a community that can create services based on Yahoo! Pipes. Anyone has any ideas on how effective something like that maybe??
Anyone have an example of turning a phpBB (or any) forum into an RSS feed? I've got a favorite forum that's not available via RSS/Atom/etc. It looks as though Pipes might make it accessible for me but (at least at first glance) I can't figure out how!
Fascinating and quite well implemented. It isn't for joe public yet but it should help the many programmers out there who aren't RSS aware become RSS aware.
