Yahoo! Pipes For far too long now RSS has been used in ways that don't really tap its true potential. Being able to syndicate my favorite headlines or blog posts is great. In fact, it helped to kick off a revolution in personal on-line publishing that is still growing and evolving. But I want so much more.

It's not for lack of vision. Back in 2005, Adam Bosworth painted a vision that eventually manifested itself as GData. I wrote about that a bit in Google's GData, MySQL, and the Future of on-line Databases.

It's not for lack of data either. You can get RSS output from lots of non-news and non-blog stuff. Everything from classifieds on eBay and craigslist to Bugzilla, Wikis, and so on.

The problem has been a lack of good tools for pulling it all together. In the Unix world, we often connect sources of data to filters and utilities using pipes. A pipe is a way of constructing ad-hoc workflows composed of any number of inputs, filters, and manipulation tools. And the beauty of the whole system is that they all use a very simple input and output method, so there's a nearly infinite set of ways you can combine and recombine them.

On the web, however, it's harder. There are data sources and feeds, but until now we've had no pipes! Pulling together and integrating data sources using JavaScript isn't on the client for the faint of heart. The browser isn't the same as a Unix command-line, so building mashups has been more frustrating and time consuming that it needs to be--especially for Unix people like me.

Yahoo! Pipes is one of a very small set of completely amazing on-line data manipulation and data mashup environments that can really change the way we work with on-line data sources. (The others are DabbleDB and Dapper.)

Yahoo! Pipes is...

Pipes is a hosted service that lets you remix feeds and create new data mashups in a visual programming environment. The name of the service pays tribute to Unix pipes, which let programmers do astonishingly clever things by making it easy to chain simple utilities together on the command line.

So if you're interested in an interactive on-line data mashup construction set, check it out and have a look at the existing pipes developers have created. I think it's one of the coolest web apps we've released in a long time, and I'm not just saying that because I know the guys who built it. Not only does it make hard things easy (like Perl has done for years), the user interface kicks ass too. It's exciting both from a technological point of view and because of the implications for the web as a whole. :-)

For another great take on this, check out what Tim O'Reilly has to say about Pipes. I don't think he's over-selling it by calling Pipes "milestone in the history of the internet." It's seriously cool stuff.

I'm hoping we can pull together some demos and interviews with the team next week. Watch the Yahoo! Developer Network blog for more...

See Also:

Update @7:20am on Feb 8th: Pipes is currently down. There's a lot of interest, so things are getting tweaked on the backend. I'll post more when we have an update.

Update @7:00am on Feb 9th: Pipes is alive and tuned! Try it out.

Posted by jzawodn at February 07, 2007 08:06 PM

Reader Comments
# Matt Cutts said:

Seems like a really cool idea! I tried it and got "invalid module type: pipe:GipCBi.32xGsFE3JXqIxGw at /home/y/lib/perl5/site_perl/Yahoo/RSS/Pipe.pm line 369." I was trying to hook a yahoo search up to the pipe output.

on February 7, 2007 08:47 PM
# Brett said:

Very interesting. Today I unsubscribed to IMDB's daily news feed because I'm only interested in the movie review summaries they do every Friday, and it wasn't worth it for me anymore. But, it took me about 10 minutes to develop a pipe that returns an RSS feed of just those reviews. It'll break if they change the way they name their articles, but it's pretty damn impressive nonetheless.

on February 7, 2007 08:50 PM
# Jeremy Zawodny said:

Thanks, Matt. I'll make sure they see the bug.

This stuff is, of course, beta right now. :-)

on February 7, 2007 08:51 PM
# DeWitt Clinton said:

Congrats, Jeremy!

I think you guys have a winner here.

Best,

-DeWitt

on February 7, 2007 08:55 PM
# Ram Prasad said:

Wow. Excellent App. Endless possibilities .. I love the UI

on February 7, 2007 09:07 PM
# Edward Ho said:

Thanks Matt!
I believe we've got that bug fixed now, please give it another try.

on February 7, 2007 09:09 PM
# Glenn Slaven said:

Very cool service, but I'm getting an error every time I hit the site now:
"Sorry, Unable to process request at this time -- error 999."

I'm guessing this is just due to server load?

on February 7, 2007 09:10 PM
# Jeremy Zawodny said:

Nope, that's not related to server load (usually). I'll shoot you an email to see if we can figure it out.

on February 7, 2007 09:14 PM
# Matt Cutts said:

Looks much better, Edward Ho, thanks! Can you say a word about when pipes are public/private? Maybe saving a search = get an obfuscated url, and publishing it makes it publicly viewable?

Also, do you have to be Yahoo-ID-logged-in to run a pipe, or do you just need to load the obfuscated url?

on February 7, 2007 09:45 PM
# Edward Ho said:

Matt:
Sure, so pipes are all public in the sense that anyone with the obfuscated url can run and subscribe to it. We did this to facilitate sharing via IM/Email/etc...

You do not need to be logged into Yahoo to do any of the above. You do need to log in if you want to clone a version to modify for yourself.

The "Publish" feature is to make your pipe visible to others via Search and Browse.

on February 7, 2007 09:55 PM
# someMadeUpCrap said:

the editing ui is very cutting edge, pretty much the limit of what you can stuff into a browser. maybe a little over the limit, it killed firefox2 on freebsd62 for me. but thats fine, if you are going to experiment, go to the frontier or don't bother.

not sure how many people will use and rely on the actual pipes, though that also likely does not matter.

kudos to the team for trying something beyond the triviality of most "mash" initiatives.

on February 7, 2007 10:33 PM
# Justin Watt said:

Using "RSS" as a generic term for feeds and saying that Perl makes "hard things easy"...?

on February 7, 2007 10:54 PM
# Jeremy Zawodny said:

Yup.

RSS == Feeds, whether they're RSS x.y, Atom, etc. (at least in my head)

And Perl did make hard things easy for a whole lot of the first generation of the Web.

on February 7, 2007 10:57 PM
# Panayotis Vryonis said:

This is amazing! Needs some work, ex. problems with unicode characters, but the whole implementation is great.

Finaly my dream (http://vrypan.net/log/2005/10/31/rip-mix-burn-for-feeds/) comes true...

Now, could you please buy FeedBurner and intergrate their services into Pipes, please?

on February 7, 2007 11:32 PM
# Jeremy Zawodny said:

Heh. You pony up a few hundred million dollars, and I'll see what I can do. :-)

Gald you like it, though.

on February 7, 2007 11:41 PM
# Nik Cubrilovic said:

"Heh. You pony up a few hundred million dollars, and I'll see what I can do. :-)"

Sounds like you guys and FB have been talking :)

on February 8, 2007 12:17 AM
# Andreas Gohr said:

That's great. I just played around with and I love it.

But (of course there is a but) it would be great if there would be a sitescraper source could be added to extract data from websites without feeds. You probably didn't do it for IP reasons, I guess.

What I couldn't find is a way to manipulate items of a feed but maybe I haven't looked hard enough. Eg. I mixed several feeds into one and wanted to add the source of an Item to the title...

on February 8, 2007 01:25 AM
# Feng Qu(A prod Yahoo! employee) said:

Life is a pipe...

on February 8, 2007 01:25 AM
# Feng Qu(A prod Yahoo! employee) said:

Life is a pipe...

a proud Yahoo! employee

on February 8, 2007 01:27 AM
# eran shir said:

Pipes is really cool. And what's more its UI is fun and useful at the same time. An as you go preview would be very helpful.

Next stage is to populate the sources options with Dapps (i.e. with all of the web) :)

Kodus to Yahoo on bringing a real innovation

on February 8, 2007 02:46 AM
# Karl said:

Wow. This is the *perfect* tool for me to illustrate to my coworkers a number of concepts relating to blogging, rss, and the universal database of participation all of us have been building.

Sure you could all of what Pipes provided with some Python or Perl fairly quickly - if you were a scripter like myself.

But this opens it up to almost anyone. And that makes it huge.

On a personal note, this is precisely the kind of tool that will enable me to provide useful views into the conversation talking place in the Philly Future regional blog community.

I don't see an OPML input there, but I've just dipped my toes in this morning.

Great work Yahoo! :)

on February 8, 2007 04:04 AM
# Hatem said:

Pretty original, I love it ! I posted a quick review on AJAX magazine too.

on February 8, 2007 04:21 AM
# Bruno said:

Good to see Yahoo innovating.
No doubt that we are at the beginning of a new area http://brunovernay.wordpress.com/2007/02/08/yahoo-pipes/

At least, that kind of tool may drive the adoption of more "processing friendly" techno, like XHTML and other Semantic-web stuff that only geek seems to care about now.

on February 8, 2007 05:49 AM
# Dom said:

Sounds like a fabulous service... aside from the fact it's totally dead at the moment. :)

on February 8, 2007 07:10 AM
# Reggie said:

I second Dom. I've been able to actually access 'Pipes' maybe 1 in 4 instances. The only time I was able to actually get in, I got a 'System Error' notice.

on February 8, 2007 08:18 AM
# Jonathan Trevor said:

We're working on it...

Jonathan

on February 8, 2007 08:18 AM
# Peter Cooper said:

Finally, real competition! With FeedDigest we've been doing the generation before this for almost two years now, and now it's time to kick things up a notch or two, which Pipes appears to do. Of course, competition brings the big guns out, so it'll be interesting to see what happens next ;-)

on February 8, 2007 08:52 AM
# Scott said:

Using RSS, that's a start. How about allowing us to define data types || sources or take a Ning-like approach and let us clone existing sources and modify them?

The first new data type I could think of would be JSON.

on February 8, 2007 09:08 AM
# Joe Duck said:

Yahoo! This is brilliant. Pipes stands squarely in the middle of where it's all going.

on February 8, 2007 09:24 AM
# mb said:

This is what I'm seeing on the pipes pages right now: "Our Pipes are clogged! We've called the plumbers!"

Popularity is a high-class problem! Congrats!

on February 8, 2007 09:42 AM
# Jake Kaldenbaugh said:

Isn't Pipes sort of like Excel for internet data feeds?

on February 8, 2007 10:55 AM
# Jeremy Zawodny said:

No, DabbleDB is Excel for the Internet.

on February 8, 2007 10:57 AM
# Vignette BIS said:

Say, this is "Pipes" thing is same product as Vignette's BIS. The main difference is BIS uses a Windows client and Pipes is web-based.

http://www.vignette.com/us/Products/Enterprise+Content+Integration/Vignette+Business+Integration+Studio

http://www.vignette.com/dafiles/images/IM0601_ss_BISLg.gif

The quick GAP analysis:

1) Pipes should let a user interrogate a WSDL file. Pipes should then present to the user the fields defined in the WSDL required for input and let the user map the resulting outputs.

2) Vignette BIS integrates with a product called Vignette Builder:

http://www.vignette.com/us/Products/Portals/Vignette+Builder

http://www.vignette.com/dafiles/images/IM0601_ss_BuilderLg.gif

Builder takes the Pipe (err ... BIS output) defined by BIS and lets the user create forms and tables based on the inputs / outputs.

Conclusion: Pipes should let users create rich Portlets (err...WIDGETS) based on their Pipes and publish them to My Yahoo or other blog. Oh wait, it already does!

3) JDBC: BIS lets users utilize a JDBC data source. Easy to add to Pipes.

Not so revolutionary fan boi's, but I'll say that Vignette is screwed and wasted a good piece of tech.

on February 8, 2007 11:35 AM
# Jrom said:

Nice, the pipes are clogged. Doesn't bode well
when Yahoo cannot even maintain uptime with
this.

on February 8, 2007 11:59 AM
# Jay L said:

Just wanted to link to another ITconversation download: Part II of Robert Lefkowitz's Semasiology of Open Source. I highly recommend the whole 3-part series, but part 2 is facsinatingly relevant to what Tim O'Reilly is talking about when he says "mashup programming to the non-programmer", and also some of the other takes on Pipes I've read around the blogosphere today.

http://www.itconversations.com/shows/detail662.html

on February 8, 2007 12:52 PM
# Jonathan said:

Yeah, clogged for me too.

Makes me wonder: why can Google and Yahoo! not prepare well enough for such user demand? Both companies know the demand exists, they should both be well equipped for the service.

Happened with Google Analytics, Gmail, name it you got it...now Pipes too? :(

on February 8, 2007 01:33 PM
# Rasmus said:

Pipes are back.

on February 8, 2007 08:02 PM
# Serollah said:

And I was just about to turn off my computer and go to the gym .. good grief, will the fun toys to play with never stop?

on February 8, 2007 10:44 PM
# Olivier Travers said:

Exciting and promising stuff. I hope Yahoo is going to clarify whether this tool is intended to target only end (power) users or if it's going to be supported in a way that makes it "serious enough" to be used by companies too (think, say, Bloglines vs. Amazon S3).

on February 9, 2007 02:44 AM
# Bob Afifi said:

I can't figure it out :-(
I just want to be able to re-combine a bunch of RSS feeds and create into one like I can with FeedBlendr which unfortunately does weird things with apostrophes.. Can somebody please make a tutorial or show the steps?

Thanks,

-Bob

on February 9, 2007 09:00 AM
# Anon said:

Are VoIP Bloggers original ?
A fun take using Yahoo Pipes here:
http://corporaterat.blogspot.com/2007/02/piping-hot-are-voip-bloggers-really.html

on February 9, 2007 09:44 AM
# av said:

Jeremy, Yahoo Pipes folks:

Great idea. Very good implementation. Like any IDE, shouldn't there be some way to get help? I was trying to build a pipe but couldn't figure out some details - a help function would have been helpful.

-Anupam

on February 9, 2007 02:53 PM
# Chad Lindstrom said:

It looks like functionality right now is a little broken as no pipes seem to output much right now.

Besides that... this is some great stuff. Who would have thought... pipe and filter. Now why didn't I think of that!

Great job Yahoo!

on February 9, 2007 11:35 PM
# Harish TM said:

Trying to build a community that can create services based on Yahoo! Pipes. Anyone has any ideas on how effective something like that maybe??

on February 11, 2007 12:35 AM
# Kevin Roth said:

Anyone have an example of turning a phpBB (or any) forum into an RSS feed? I've got a favorite forum that's not available via RSS/Atom/etc. It looks as though Pipes might make it accessible for me but (at least at first glance) I can't figure out how!

on February 11, 2007 10:55 AM
# franco said:

Fascinating and quite well implemented. It isn't for joe public yet but it should help the many programmers out there who aren't RSS aware become RSS aware.

on October 15, 2007 07:41 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.

 

Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.