I'm on a quest to find the perfect RSS aggregator. And I'm starting to feel like I'll end up having to write some code to get it. That makes me unhappy because I've got roughly a billion other things I need to be spending my time on.
I'm slowly convincing myself that I need an e-mail based aggregator. Yeah, that's right. I think I want an RSS aggregator that sends me all the interesting bits via e-mail.
The real problem is that I use several different computers and operating systems each day. And I've yet to find one that'll give me what I want. My current favorites are AmphetaDesk and NetNewsWire. The former is a client-server application (I have it running on a co-located server so I can access it from anywhere) and the later is a GUI application for Mac OS X. For me, they're two "best of breed" applications.
So, let's look at the general cases.
The Good
The Bad
The Good
The Bad
And now the specific cases:
The Good
The Bad
The Good
The Bad
I'd like to combine aspects of both models. I'm thinking of a server-based process that can gather all the data and give it to me in one of several ways. Maybe I can just point my browser at it and catch up on the news--just like AmphetaDesk. That's great for when I'm on-line and in a surfing mood.
I'd like it to do RSS auto-discovery.
I'd like the option of having updates sent to me via e-mail and possibly instant-messenger. Heck, I'd like to be able to subscribe via e-mail or IM as well.
By having an e-mail option, I can work off-line whenever the mood strikes me or connectivity vanishes for some reason. And if it provided a XML-RPC or SOAP-based API, I could imagine having a nice selection of desktop GUI apps that can provide a nice interface to the data--on any platform (Linux, OS X, Windows).
Anyone know of such a beast? Sounds like it'd be a fun project to build. It breaks down nicely into logical units, I think. There'd the be the core server. It polls all your feeds, figures out what's new, and so on. Then there's an API for subscribe/unsubscribe and reading content. You could then have clients or bridges that speak Jabber, XML-RPC or SOAP, POP or IMAP, etc.
Hmm. If only I had some more spare time. This is starting to sound really fun. Heck, it'd even be a good excuse to learn one of the languages I've been wanting to get into: Python or Ruby.
Sigh. Back to doing work.
As of this moment, if you search google for "jeremy", you'll find that my weblog is the first link returned. And, better yet, there are no ads or sponsored links on the page.
I wonder how long this will last...
I just watched Barnes, one of my cats, catch and eat a fly. I have trouble killing flies with a fly swatter, but he was able to do it with a swipe of his paw. Wow.
According to Business 2.0:
Entire PCs can now be bought for less than the cost of one copy of Microsoft Office software -- and an interesting new market is born.
Hmm. Suddenly it's clear that Microsoft might be getting a bit too expensive for the low-end market.
Some of the world records for push-ups are pretty amazing. Paddy Doyle did 1,500,230 in a single year. Minoru Yoshida did 10,507 non-stop. There are some pretty insane folks on that list.
This is most excellent news. The more I learn about Apple and OS X, the more I see that they're really "getitng it."
I've been trying to keep track of all the things I wish I had known about the TiBook before I began using it on a semi-regular basis. Nat's weblog entry got me thinking about putting my list on-line, so here it is. Some of these things are surely due to my lack of figuring things out, but the Mac is supposed to be easy to figure out, right?
Reboots - I know that OS X "is Unix" under the hood, but for a Unix box I find myself needing to reboot it way too often. It seems that nearly every software update I get requires a reboot. What's up with that?
Screen Resolution - My ThinkPad T23 has a nice screen. It does something like 1440 x 1050 resolution quite nicely. I'd have thought that this beautiful TiBook screen would be capable of a little more than 1152x768. Sheesh.
External Displays - When I plug in an external display, there's no way to make the menu bar appear on it. This is really frustrating when I plug in my 21" Philips monitor. The TiBook can drive it at 1600x1200 (or even higher, but this is my preferred resolution on a 21" monitor). So I have to drag the pointer back over to the comparatively little TiBook screen just to pick a menu item. This is really, really dumb. Either I'm doing something wrong, or someone at Apple forgot to engage their brain on this one.
Connectivity - There's no obvious way to connect to the Samba server running on one of my Linux boxes. Why would I want to do that? Because most of my Office documents are stored there. I suppose I could install WebDAV on the Linux box and treat is as an iDisk, but why should I have to do that?
Power Management - I'd like to be able to close the lid on the TiBook and NOT have the machine go to sleep, lose all network connections, and so on. I'm a bit puzzled as to why there's no option for that. What if I want to close the lid while a CPU-bound task runs all night?
Point-To-Focus - There's no way to get "point-to-focus" in the Mac GUI. Heck, even Windows lets you do that if you install TweakUI. Anyone know if this is fixed in 10.2?
Tip of the day. When there's network maintenance going on, don't try to test your work remotely (like from home at 11pm on a Friday). Why? You application will behave as if it's on crack. You'll wonder why. You'll debug. You'll become completely mystified.
Then you'll remember the announcement from earlier in the week. The one about the pending network changes on Friday night.
*sigh*
At least I didn't waste too much time.
It must be time to go read weblogs.
As much as I bitch about slashdot being a pile of crap (not the technology or the people behind it, but the idiots who have nothing better to do than "first post", "natalie portman's titls", and "Linux rulez! Microsoft/Apple/Sun suck", there's some good comments to read today. Of course, I read with a threshold of 4, so I tend to "miss" all the real crap.
Anyway, here are the stories: How Should You Interview a Programmer?, Mac OS X Switcher Stories ( which points to Tim's article), and Mac OS X 10.2 "Jaguar" Reviews Pour In.
I'm trying to get back into riding on a regular basis. I came home a bit early from work last night and went for a 45 minute ride. It felt great. So I convinced myself to do the same this morning before work. I'm so glad I did. I fell a lot better during the day after I've had a ride in the morning.
The weather this morning was excellent. It was just cool enough that I wore a light jacket when I left my apartment--must have been 56-58 degrees out. Clear skies and nice warm sun. Once the ride was about half over, I was warm enough to take it off and just enjoy the weather.
I've gotta keep doing this until the rainy season starts...
New rule. I no longer read or respond to e-mail that doesn't have a Subject. If someone can't take the time to briefly summarize that their message is about, why should I take the time to do it?
Okay, I know of a few OS X folks who read my blog. Can someone tell me how to get point-to-focus rather than click-to-focus on OS X? I have it on Linux and Windows and find it rather annoying to work in a click-to-focus world. If I could get significantly higher resolution screen on the TiBook, it wouldn't be so bad. (My ThinkPad has a nice high-res screen that gives me a lot of room.)
I've tried various google searches (web and groups), but can't seem to come up with the right keyword combination to get the answer.
David Brooks has an excellent 2-part article (part1, part2) in the Weekly Standard, called "Patio Man and the Sprawl People." It's quite long but well worth the read. I won't bother summarizing it here, but reading it made me think of some of the friends I made in college. Many are now married and moving into nice, new, shiny suburbs. (Hi, Dan.) Meanwhile, I live in the armpit of Silicon Valley and am not sure where I'd really like to end up someday.
I was reading this enlightening entry (several folks are already pointing to it in their weblogs) in Ray Ozzie's weblog when I realized that Open Source software has much of the same potential for leverage as building software on Windows. (I never gave much thought to how hard it must have been to build desktop apps when I was a teenager...)
Well, my first thought after reading it was "gee, he just explained a lot of why I can be so productive with Perl on a Linux/Unix box." By using the tools that are always there and some code from CPAN, I can build relatively sophisticated applications. Those other tools are really platform technologies--things like Apache as the web server, MySQL/InnoDB, and any sendmail compatible MTA. (It's interesting to notice how virtually every popular MTA is "command-line compatible" with sendmail. An effort to commoditize sendmail itself? Maybe.)
There is a lot of what Ray describes going on in the parts of the Open Source world that I tend to inhabit--mainly the server side and middleware as well as scripting. In my job at Yahoo, most of the code I work on is behind the scenes stuff that end-users never interact with. Recently I've been working more on a user visible stuff, but it is still on the server side.
That made me happy. He was affirming what I and many others do for a living. We're building software efficiently by reusing code and components.
But none of what I do is in the space of desktop applications, but that that's where Ray's story kicks in.
On the "desktop," Linux has a lot of well understood problems. However, I think they really run a lot deeper than some folks understand. The fact that we have both GNOME and KDE, for example, isn't great when it comes to "conquering the desktop" but that's a side issue. Look at the applications, not the window managers and eye candy.
As Jon Udell pointed out in Byte (back in 2000), summarizing Miguel de Icaza:
major Unix applications such as Samba, Apache, ftpd, and ssh share essentially no code beyond the primitive facilities in the standard runtime library, libc.
I know what you're thinking--Samba, Apache, and so on... They're all server-side applications. Hang with me.
He brought this up again in 2001 in his "Embrace and Extend" article in Linux Magazine (go read it). His points are still valid today. Linux application developers haven't built the necessary cultural infrastructure to enable the level of component reuse needed to make Linux an appealing platform for application developers.
Things have come a long way since Ray Ozzie had to worry about:
having to write a ton of junk from scratch e.g. custom memory managers, code component managers that did all sorts of trickery for memory conservation, multi-byte character handling, rich text editing, MDI window handling, and all sorts of grunge
You get some of that when you select the right libraries. By using glibc and Qt or [pick your favorite GUI toolkit], you can assemble the beginnings of a foundation--the sort of foundation that Microsoft has provided developers for years in the form of the Win32 API.
However, when you begin to move up to the next level or two, things get shaky again. How should your application store configuration data? There is no Registry on Linux. So the developer has to build that. You could use plain text file (common on Linux), XML (common in Java), or something else.
(Speaking of Java, a lot of this isn't an issue for Java developers. But that's largely because Java itself is a platform full of moderately well thought out APIs. Sun wasn't stupid. They know what it takes to build applications. Of course, Java on the desktop has been less successful than on the server side... But that's a whole 'nother discussion.)
What about a common database API? That depends on the language you're working with. Python? PHP? C++? Java? Perl? They're all different. There really is nothing like ODBC, ADO, or OLEDB on Linux/Unix that has much traction.
What about printing...?
The list goes on. There's a lot of common plumbing that's not there. Not like it is in Win32 (or Java) at least.
What am I getting at? Basically, I'm agreeing with Jon. But this time I've arrived at that conclusion in a why that surprised me. I realized how much of a different world it is on the desktop vs. on the server when you're using Linux or Unix (and leaving Java out of the picture) by thinking about why I never build desktop apps (even for fun). I could probably still do it on Windows if I had to. But it'd be quite a battle for me on Linux.
It's hard to explain this to folks who haven't experienced it before. But in my previous job, I lived in the loving clutches of the Microsoft world. I built mostly server-side code for our corporate intranet. I used Perl, ASP, VB/VBScript, Access, and Oracle. But on occasion I needed to build desktop applications--quick interfaces to something or a prototype. Amazingly, I was able to re-use a lot of the tools, knowledge, and APIs from the server-side and middleware development when it came time to build desktop apps. It's really not like that in the Linux/Unix world. Until it is, there may not be significant commercial interest in building applications for Linux desktop.
Is .NET and SOAP the solution as Jon suggests? No, it's not the solution, but it may be part of the solution (or a solution). This really isn't a technological problem. It's a cultural problem. Open Source is good as solving technical problems. If that's all it was, we'd have solved this years ago.
So what do we do?
It appears that QtMyAdmin is a nice desktop version of the popular PHPMyAdmin, used to administery MySQL from a nice GUI interface.
Aparently there's a good-sized market for black socks that you can order on-line. I learned this by reading The Great Socket Experiment thread over on K5.
Give this to every Outlook Express user you know. They need. All of them.
According to an article at the Register, they're making software:
The software, called Levanta, is designed to reduce the cost of administering virtual Linux servers on IBM Corp eServer zSeries mainframe environments. Scheduled for general availability in October, the software is being demonstrated for the first time at the LinuxWorld Conference and Expo in San Francisco next week. It supports IBM's z/VM virtualization software version 4, releases 2 and 3, as well as SuSE Linux AG's Linux Enterprise Server 7 with kernel version 2.4 and Red Hat Inc's Linux 7.2. It is scheduled to cost from $150,000.
Yup, they're making mainframe software now. What an odd twist of fate.
So we're getting ready to launch some redesigned stuff on Yahoo! Finance and are taking the CSS plunge (finally). Even ignoring the ancient Netscape 4.x browsers, it's not easy. Looking at Galeon/Mozilla on Linux and Mozilla/IE/Opera on both XP and Mac OS X (oh, and OmniWeb on the Mac) highlights a lot of places where they don't agree. I wish I had the time to take screen-shots of each and post them. Hmm. Maybe tomorrow.
Yikes, I guess we ought to look at Konqueror too. What other browsers GUI are we missing? Yeah, I know about lynx, links, and w3m. Let me know if we're missing some.
How do others deal with this? Fall back to the tried and true table-based layout for some of the tricky items?
Oh, this will also be one of our most visible "live" uses of MySQL. More on that later.
If you'd like to see the layout in your favorite browser (one we haven't tested, maybe), let me know. Your help would be appreciated--especially if we can get a screen-shot out of the experience. (I'm way behind on the rest of my life this week as a result of all this stuff. Maybe I'll blog that story someday too. There are some important human lessons buried in this experience.)
Update: It's almost 3am again. I should really get to bed. *sigh* So much mail to catch up on...
If I ever get the spare time (after the book?), I'll hack my Tivo to do this too.
This thread on a RackShack support forum is amusing. Users are explaining how to remotely replace the OS that RackShack gives them with Debian GNU/Linux. Thanks to the Trained Monkey for the link.
Bruce Sterling attempts to correct Eric Raymond's "cathedral and bazaar" metaphor with something more accurate.
It's this classic artisans-versus-factory model. It's not about a bazaar. Because bazaars are pre-industrial, they're swarming with crooked rug merchants, and pickpockets, and lepers straight out of the Arabian Nights. Open Source isn't about being some kind of canny rug merchant with an eye out to make some fast dough. Open Source, basically, is about hanging out with the cool guys.
I think he's right.
How cool, you can get an RSS feed from RateYourMusic.com. I was on their home page page when I noticed the orange XML icon at the bottom. I clicked it and was pleasantly surprised to find an RSS 0.91 file streaming at me.
I'm confused. According to this Slashcode FAQ entry>, I should be able to pull an RSS feed for any user's journal. But I'll be damned if I can make it work. I keep getting a redirect when I try the URL the docs say to use.
Anyone know what magic incantation I'm missing? According to the docs, I ought to be able to get Matt's journal feed by using this url:
http://use.perl.org/journal.pl?op=rss&uid=Matts
But, as I said, it redirect me to this url instead:
http://use.perl.org/search.pl?op=journals
Grr.
If you do a view source on any journal entry or journal index page, there's no obvious link tag pointing to an RSS feed either.
I figured something like this existed but hadn't seen it until now. Apparently they're rather inexpensive too. So if you wanna put some old devices (like your SliMP3) on a wireless network, this is the way to go.
While reading something somewhere (funny how you can easily get lost reading weblogs), I ran across a skin (replacement template, really) for AmphetaDesk, my preferred RSS aggregator. It fixes the one major beef I have with it: the fact that the size of the main page increases a lot as you add new feeds. The new template adds a nice outline mode.
I'm now running AmphetaDesk 0.94 from CVS with the new template and loving it. But I'm drooling to try this out.
If you've got a PocketPC, check this out.
PocketFeed is an RSS/RDF news aggregator that runs on the Pocket PC 2002 PDA's. Using OPML for it's blogroll, you can now take your news, blog entries, or any other type of syndicated data with you, sync it wirelessly or read it offline. So yes, you can sit in Starbucks with your PDA and read the latest news over 802.11b. If you have any questions, comments or suggestions, please feel free to use the PocketFeed mailing list.
Anyone got a PocketPC that I can, uh, borrow or review? :-)
Or does anyone have something similar for a Palm m105?
I spent a fair amount of time of Friday trying figure out why our FreeBSD servers running MySQL 4.0.2 were doing so much better than our Linux servers running MySQL 4.0.2. They're all slaves of the same 3.23.51 master and get roughly equal query loads, thanks to our Alteon load-balancers (yes, the ones that occasionally stop working right).
What I noticed while watching each of them with mytop is that the Linux boxes seem to have far more slow queries than the FreeBSD boxes. Now the FreeBSD boxes in question are newer. They're Compaq DL-380s with dual 1.2 GHz CPUs, 2GB of RAM, and 6 36GB SCSI disks. The Linux boxes are a bit older and slower. But the difference was still surprising. Over the last 24 hours, the FreeBSD boxes had each logged 3 slow queries, while the Linux boxes had logged a few thousand of them. Clearly something was up.
So I got on the boxes and noticed something odd. The load average on the Linux machines was higher than I'd expect. Rather than being in the 0.5 - 2.0 range, it was hitting between 7 and 9 during busy times. Odd. I ran top for a while to see if I noticed anything odd. Sure enough, after a few minutes, I found the pattern. The kswapd process was using up a fair amount of CPU time--sometimes as much as 99% of one CPU.
It gets more interesting. Both Linux boxes have swap disabled. It's been that way ever since I got sick of dealing with the 2.4 kernel's brain-dead virtual memory system last year. Why would kswapd even be running on a system with no swap? I have no idea.
But I decided to do some research and see if anyone had seen this before. The closest I got was this message on the linux-kernel mailing list, a complain by MySQL AB's own Sascha Pachev.
He noted similarly odd behavior and asked that Rik look into it. Unfortunately, I haven't been able to find any follow-up messages.
So I went back to looking at the configuration on the machines in question. Both have 2GB of RAM, roughly half of which is for MySQL. I have the key_buffer set to 512M as well as the innodb_buffer_pool. That leaves 1GB for the OS cache, buffers, and related stuff. It should be more than enough, shouldn't it?
Just for the heck of it, I backed both values down to 384M and restarted MySQL. After an hour or so, things began to look bleak again. Lots of slow queries and the kswapd process (actually a kernel thread) was getting more CPU time than I'd like. It was at this point that I really began to marvel at the situation. The FreeBSD VM subsystem never does stupid things like this. In fact, our MySQL/FreeBSD boxes rarely swap unless I do something really stupid. How can the one in Linux be this much worse? Beats me.
Anyway, even more frustrated, I decided to re-enable swap reboot the machine. At this point, I had little to lose. Once it came back up and I got MySQL started, things looked okay. kswapd wasn't as busy, and there were fewer slow queries. In fact, after 1 day and 9 hours, the server has only logged 66 slow queries. But according to top there's about 47MB of swap in use. The resident size of mysqld is 736MB, while it's overall size is 816MB. Apparently the kernel swapped out part of the buffer pool for InnoDB or the MyISAM key buffer.
I guess that extra gig of memory isn't enough for it.
I fail to understand what it's doing. But the machine seems to perform better with swap enabled. The only theory I've developed so far goes like this: With swap disabled, the kernel (being very stupid), goes looking for pages that it can swap out. It finds them but cannot swap them to disk. Next time around, it repeats this process, never realizing how futile it is. With swap finally enabled, it can swap out some memory and get the breathing room that it thinks it needs.
If anyone has hints on how this can be tuned (like telling the kernel not to bother), I'd LOVE to hear about it.
Linux may have FreeBSD beat when it comes to threading, but it sure could learn a lot from FreeBSD when it comes to virtual memory management.
Update: Thanks to the folks at NewsForge, you can find a teaser for this blog entry here (and it's currently on the home page). They picked this one up quite fast. I'm impressed.
Update #2: Allow me to respond from some feedback that I've seen so far. First off, we've been running 2.4.18 for quite a while now. We started with 2.4.9, tried 2.4.12 and 2.4.16. There's only so much time I can spend switching kernel versions and re-testing. Now that 2.4.19 is out, we'll give it a shot.
A few folks have suggested that since FreeBSD is the best tool for the job, I should just shut up and use it. If only that was the case. I'll post another entry in a few days detailing the problems with running a high-volume MySQL server on FreeBSD. It has issues of it's own, mostly related to FreeBSD's poor threads implementation.
Thanks for all the feedback so far. Some of it looks promising. The flames, however, are simply ignored.
Find it here and install it if you use Windows and IE. Works pretty well. Adds an entry to the "Tools" menu in IE so you can customize it. And best of all, it's free. :-)
I've been doing a lot of catch-up reading at K5 recently. I found the Liquor Guide to be particularly helpful. It's only a basic introduction, but there's a lot of good discussion around it and I hope it'll lead to more stories.
Michael Brewer has a good article on O'Reilly's Mac DevCenter that covers .Mac pretty well. It also links to Rael's article about WebDAV on Mac OS X, which I found very useful.
Simon has hit on something that really bothers me about the "Perl Community"--the way it often rewards arrogance and spwans cults based upon those in the inner circle. It's sad to see Simon go, but it's clearly better for him this way.
Earlier, Dan pointed me at this amusing story at Recoil. I had never heard of Recoil before, but it feels a lot like The Onion. Give it a read today.
Maybe you've switched and are now wishing to have the Alt-Tab feature present in the operating system you left. Have a look at Lite Switch X, it's probably just what you need.
I fail to understand why some folks make it difficult to find the URL of the RSS feed for the weblog. Take, for example, Simon Cozens. Do you see a link anywhere for the RSS? Nope. If you view source in your browser, you'll see it near the top. But why not put it where humans can see it too? You know, the kind who want to cut-n-paste it somewhere?
Another example is all the weblogs hosted on O'Reilly Net, like Andy Oram's or Tim's. Where's the RSS feed? Do they not get it? My RSS software wants URLs to watch for headlines...
I have to agree with the Fuzzy One. Dave is on crack. Open Source is not slavery.
A helpful article at LinuxOrbit. I've always thought the process was a bit under-documented. Luckily, I don't really need to build custom kernels for my Debian boxes (except my laptop, which I have no plans to update).
But I don't know where. It's either FreeBSD, LinuxThreads (on FreeBSD), or MySQL 4.0.2. Grr. Lots more testing required. Luckily, I've figured out how to reproduce it pretty well.
Why me?