Jeremy Zawodny's blog

Recent Sphinx Updates

If you use the Sphinx search engine and have been watching the development branch (0.9.10) and wondering when to upgrade, I'm here to tell you that "now" is a great time. As of r2037, the last major issue I regularly saw has been fixed. The other big bug was fixed in r2031.

Late last week I began testing those fixes in a "burn-in" test I've developed that makes liberal use of indextool --check. Instead of seeing index corruption within an hour, I saw none. After 3 days of no failures, I deployed it to a subset of our search back-end servers. Yesterday we deployed it to half of the remaining servers.

So far, so good!

I should note that all our index corruption was merge related. Sphinx wasn't building corrupt indexes out of the box, but the merges (usually filtering merges) could produce corrupted indexes.

We were upgrading from a lightly patched version of r1894. That meant rebuilding our indexes to use the new and more compact format. Some of the obvious benefits of the upgrade:

• smaller disk and memory footprint
• pre-fork support to spawn searchd children at start up
• more reliable shutdown and pid file handling
• kill lists
• mysql protocol support
• lots of small optimizations and fixes

Thanks to the Sphinx team for their excellent work. I look forward to the release of Sphinx 1.0.

Oracle and MySQL

In The EU and MySQL, Tim Bray treads lightly on the topic of Oracle's pending ownership of MySQL if the Sun acquisition goes through. I left a comment on his post, but he's likely to be heavily moderating what appears there since he works for Sun.

So here's what I posted on his blog.

I haven't yet seen anyone explain what motivation Oracle has for pouring resources into MySQL, especially if it eats away at their DBMS business on the low end.

I've been puzzling over this since their acquisition of Innobase Oy (the makers of InnoDB) years back. Is Oracle serious about seeing MySQL grow and succeed, or was that just a way to get a strangle-hold on a critical piece of MySQL?

I've never had the chance to ask Ken Jacobs that. Actually, I have but it would have been kind of rude. And even if I did, I'm not sure I could trust the answer.

I doubt this comment will get published, but as a MySQL long time user, supporter, advocate, and author I'm really glad to see things like PBXT, MariaDB, and Percona's XtraDB out there.

Really, we need that kind of diversity in Open Source. A MySQL/InnoDB "monopoly" wouldn't have been healthy in the long run.

A reporter contacted me today to ask, among other things, if I think Oracle was/is threatened by MySQL. Oracle claims that they serve two different markets, etc. He wasn't so sure.

Sadly, there's some background information that I should not publish here, but suffice it to say that Oracle was and probably still is threatened by MySQL. Their sales/marketing tactics made this quite clear long ago. But those deals were rarely public--for good reason.

Lazy Shopping Carts

What's with all the lazy shopping carts? It seems that through the years, shopping carts have become progressively more lazy.

Years ago, when a shopping cart was done with its duty, it had to find its way all the way back from your car to the store entrance--either just inside or just outside. But, apparently as the result of an aging population of shopping carts, grocery store designers started scattering "cart corrals" around the parking lot so that the poor carts only had to scoot a few dozen yards before coming to rest. But even that is starting to seem inadequate.

I was at Trader Joe's earlier today and noticed quite a number of carts that could only make it as far as the little curbs that surround the trees that in the parking lot. And at Safeway, where there seem to be cart corrals every 15 parking spots, I still see carts that aren't able to make it even that far.

Modern technology has done so much to our modern food system in the last 20 years (a fair amount of it quite questionable, I'm sure), yet so little has been done to help these carts! Why is that?

Who are the cruel people that insist on keeping these carts in service long after they're able to perform up to very basic expectations?

What is this world coming to?

It's sad, really.

Pass the Duct Tape

Reading The Duct Tape Programmer really hit a nerve with me. If you code for a living (or part of your living), go read it. I sometimes get annoyed by Joel Spolsky's writing, but this time he hit the nail right on the head.

I am by no means putting myself in the same league as Jamie Zawinski, but I definitely subscribe to the idea that keeping my code as simple as possible is going to pay big, big dividends in the long run.

Needless complexity really does kill you.

In my work it has manifested itself in a few ways. But one that seems to pop up over and over in recent times (and has bitten a few of my coworkers as well) is socket programming. For whatever reason, we're able to find the the freely available client libraries for interacting with some services will inevitably fail a small but non-zero percentage of the time. And when you're making millions and millions of calls or connections per day, even a 0.01% failure rate is enought to make for a really bad day.

That's especially true when the failure mode involves leaking file descriptors or needlessly long timeouts (or timeouts that fail to work properly) that result in a full or partial cascading failure. It's the ultimate in frustration.

I'm not going to name names here. Maybe at an upcoming conference I'll talk about what we've seen and how network oddities add a whole new set of failures to the mix.

Anyway, the solution ends up being one of two options: (1) forking the module and re-writing the code to use the low-level system calls instead of the multitude of abstraction layers that were supposed to make the task easier (die abstraction layer! die!), or (2) scraping the module entirely and writing your own. The first option sucks but at least you're not having to learn all the quirks of those abstraction layers. The second option sucks even more but at least you know exactly what you've got when it's all said and done.

For me it's more satisfying and productive get to know the low-level stuff well enough that you can just drop it in when you need it. It's less frustrating than trying to reproduce all the conditions and variable that cause some abstraction layer to fail.

Pass the duct tape. I've got work to do!

Watching Yahoo's Transformation

Having not worked at Yahoo in a bit over a year, it's interesting, amusing, and frustrating to watch what's going on over there. Some of their moves confirm feelings I've had for some time, while others are more puzzling.

The Search deal with Microsoft was basically inevitable. You could see even two years ago how out-gunned Yahoo was compared to Google and to what Microsoft was rumored to be thinking of doing. Based on what I knew, Yahoo was spending less than 20% of the money on search that Google was and they were trying to do the same worth with about 1/10th the hardware and less mature infrastructure software. Meanwhile, the web kept growing and became more and more real-time.

It seems that Yahoo wants to sell Zimbra. I remember when that acquisition happened. I thought it was cool technology and would be great if we actually wanted to compete in the on-line "office suite" market (or whatever), but that never happened. Instead it remained as one of those "enterprise" products that Yahoo has a history of trying and failing at. Remember Yahoo's enterprise instant messaging product? FAIL.

Now comes news that Yahoo wants to sell Small Business which includes their web hosting and domain registration businesses. This makes good sense to me. While it's a business that I think has usually made money, it simply wasn't competitive in a day and age when you can get a full virtual machine, storage, and bandwidth from any number of vendors who aren't scared to offer good remote access (ssh anyone?) to the server: Slicehost, Rackspace, Server Beach, Amazon, and so many others. Yahoo's offering may have made sense back in the year 2000 or so when it really competed on price, but this is one of those race-to-the-bottom commodity business and has been for years.

People picking on Carol for selling some stock recently? Well that's just dumb. Yahoo is a public company and she really didn't sell that much.

I hope that Carol is able to trim the parts of Yahoo that no longer make sense and help bring some focus to the company. I really do. But to be honest, I've seen it before. Terry Semel tried to do something similar when he came on board. But that was a wandering effort that ultimate lacked focus and wasn't ambitious or forward-thinking enough.

Jerry and Sue tried this when Terry left, but I really think they were too "Yahoo" to transform Yahoo into what it needs to be.

I see how Carol is trying to be smart on the business side, but I'm not sure how Yahoo plans to wow its users. And coming from the Yahoo! Developer Network, I wonder if Yahoo will ever get serious about outside developers. Conferences and Hack Days are great, but I suspect they still haven't figured out how to offer buisness-class APIs (with an exchange of money and an SLA). Aside from YUI and Hadoop, can you really go beyond a prototype with this stuff?

It could simply be that I never really drank the YOS/YAP kool-aid and never will get it.

Oh, and what about those smaller startups? Should I start to worry that del.icio.us or Flickr is going to go away? What about Upcoming? When their founders start to worry, I feel like I should too. At least Flickr has a business model and appears to still be kicking some ass. MyBlogLog has all but died on the vine, right? Is there anyone left of the original team of 5 or 6 engineers still working on it? No, I think it fell victim to Yahoo's larger social strategy. FAIL.

I hope that Carol can be clear, focused, and agressive in re-shaping Yahoo. The half-measures attempted over the last few years simply haven't been enough and never will be.

How my netbook taught me to love xmonad

I've had this low-level urge to try a new window manager for a few months now. I work on a Linux box (Ubuntu) daily and mostly run a number of terminals, GNU Emacs, Mozilla Firefox, and Google Chrome. Nothing too fancy, really. Oh, and a shitty VPN client.

Background

Most of the time I'm doing this in front of a 24" (or larger) monitor running at 1920x1200, so there's a lot of screen real estate. Yet I was always annoyed by how much time I spent moving windows around or trying to find the optimal layout--always reaching for the mouse.

Years ago when 1024x768 was the norm, I ran a heavily customized fvwm2 and enjoyed it. But then I made the move to Windows for a few years and came back to Linux with Ubuntu/Gnome as my "desktop."

The Netbook

Months ago I wrote about how I love my Samsung NC10. When I'm not at my desk, I'll often use PuTTY to login, resume my screen session(s), and continue working. For what it's worth, I find that the free DejaVu Fonts (specifically the Monospaced one) works exceptionally well.

What I realized is that my method of having one terminal in full-screen mode on each "desktop" (thanks to the VirtualWin virtual desktop manager) is surprisingly productive, even on the little 10" screen. At first I considered this a fluke and attributed it mainly to the novelty of working this way. But after a while I realized that it was the focus that this setup enforces. There simply isn't enough room to have a browser on screen to distract me while I'm coding something, reading email, etc.

I really need to focus one or a few tightly realted tasks. The cognitive overload of having the whole Internet available really gets pushed off-screen and mostly out of mind.

Trying xmonad

After a discussion in our chatroom at work the other day, I finally decided to give a new window manger a try: xmonad. A big help was Tom's Introduction to the xmonad Tiling Window Manager which gave me just the information I needed to get started.

I used it most of Friday and a bit off and on Saturday, both on my primary work computer and my "home" Linux desktop machine. The experience has been surprisingly positive so far. Most of the hassles have revolved around re-training my hands to learn some new keyboard shortcuts and finding replacements for the few GUI things that Gnome provided on my previous desktop.

On thing I particularly like is that most of the keybindings seem very sane out of the box with xmonad. I haven't really needed to customize anything yet. I have found that a couple keystrokes that I use in GNU Emacs appear to be intercepted by xmonad and I haven't found an easy way to undo that or at least discover what they're supposed to do: Alt-w and Alt-q are the two I've noticed.

I also needed to resurrect an old xmodmap file that I could use to turn my CAPS LOCK into a Control key and re-discover the right xset command to set my key repeat rate higher than the default: xset r rate 250 30.

Other than those few nits, it's been pretty smooth sailing. I definitely feel like I'll be more productive in the long run a result of switching.

Have you tried a tiling window manager? Did you stick with it?

Pandora vs. CDs

I've been using Pandora for streaming music on-line for a while now. I enjoy it so much that I pay the roughly $5-$6 per month for a higher quality stream and because I want to do my part to make sure they stay viable as a business.

What's so great about it?

Pandora is unobtrusive, easy to use, and really good at finding similar and related music that I'm likely to enjoy. I'm always amazed at how good it is when I create a new station.

Just a few weeks back, I told Pandora to create a "U2" station. A few hours later, I realized that I'd been listening to an excellent selection of tunes.

I never get bored of the music on Pandora. It's just a part of my day, always playing good music to keep me happy while I'm coding or fixing things.

I feel like an idiot for the thousands of dollars I spent on CDs years ago, not to mention all the time and effort that went into digitizing that library multiple times (my own code, iTunes, WinAmp, etc.).

If I thought I could get any real money for my 550 CDs, I'd just sell 'em all and send the Pandora crew a nice check with a note: "Keep up the great work."

If you've not yet used Pandora, I highly recommend it. You'll be surprised by how often it plays music you like, even with the most minimal input from you.

Firefox vs. Google Chrome Revisited

Last week, in Google Chrome is the New Firefox, and Firefox the new IE, I ranted a bit about how slow Firefox 3 (notably tab switching and the "awesome" bar") was on my Ubuntu 9.04 machine. Needless to say, I got some good feedback from that post and it prompted it me to do a few things.

I've since been running both Firefox 3.5 as well as the Firefox 3.6 trunk code and can say that both are notably faster than Firefox 3.0.xx. The difference between 3.0.xx and 3.5 was substantial and really helped to close the gap with Chrome. Going to the 3.6 alpha nightly builds made it even faster in some places and slower in others--not surprising since it's still in development. Scrolling was drastically worse, but I'm told that's currently in flux.

I have to hand it to the Firefox team. They're not taking Chrome lying down. I've found no real issues with running 3.5 so far and it's a bit of mystery to me (which is to say "I haven't researched at all...") why Ubuntu 9.04 isn't upgrading folks to it. I'm really looking forward to seeing 3.6 stabilize. The Javascript and layout performance seems really good in my use so far.

Google Chrome is the New Firefox, and Firefox the new IE

I spent too long on Friday screwing around with stuff on my work laptop in an effort to make Firefox's apparent performance not SUCK ASS. Ever since I upgraded to Ubuntu 9.04 I've been somewhat unhappy, mostly as a result of the well publicized issues with Intel Video on Ubuntu 9.04.

I read about possible hope with upgrading the driver which also required a kernel upgrade, so I did both and rebooted. And, as I hoped, video seemed a bit snappier.

But Firefox still SUCKED ASS.

At this point I was REALLY PISSED. Sure my new video was nice and all but making new tabs (or switching between them) was still slow, and the disaster known as the "awesome bar" (how to disable) still sucked.

So on a whim I went and installed Google Chrome. It totally rocks on my Samsung NC10 netbook (running WinXP), so I figured what not give it a try.

It turns out that Chrome on Linux is DRAMATICALLY FASTER THAN FIREFOX!.

It's been quite stable on Windows, so I'm hoping the same is true on Linux and I can just switch over to it. As of now, Firefox is my primary browser on only half my computers. Chrome seems to be slowly displacing it, just like Firefox replaced the bloated pig known as Mozilla years ago (and the long since stagnant IE on Windows).

It's funny. Browsers seem to be like Internet companies. Every few years a new, small, faster one comes along to kill off some (or all) of the previous generation. I guess this is just the latest in that constant evolution.

It'll be interesting to see how this new competition really affects Mozilla Firefox.

I spend most of my day in gnome-terminal (to screen, mutt, irssi, etc.), GNU Emacs, and a browser. When they're not fast and stable, my life sucks.

Flying an Ultralight Trike or Weight Shift Control Aircraft

On Saturday Kathleen and I took advantage of the chance to take a short class in Trike flying and then go up for introductory rides.

This was an event organized by AreoDynamic Aviation (formerly Amelia Reid Aviation) at the Reid-Hillview Airport in San Jose.

We arrived to find Pete Marsh's Antares MA-34 R912 Ranger with an 80 horsepower Rotax 912 engine sitting outside the hangar. So I took several photos of the engine, cockpit, and wing while we got a good look at it.

A view of the cockpit.

Beefy landing gear!

The Rotax 912 Engine

Glass Cockpit

After checking it out, Pete spent about an hour telling our class about Trike flying in Alaska, answering our questions about the sport and the machines, and giving us a little bit of an idea what to expect.

For the record, this Trike has electric trim and will cruise at about 70-75mph while burning 2.5 gallons of high octane auto gas per hour. So mileage is comparable to most cars. But the ride is a hell of a lot more fun! And with a 15 gallon fuel supply, you can fly a lot longer than your bladder will let you.

Several of us had reserved times to fly 1 hour $99 introductory rides with Pete, and before I knew it my time had arrived. I was really looking forward to the experience.

Pete helped me get strapped in and I took a few pictures and a vide as we taxied over to the run-up area for runway 31R.

My turn!

Strapped in and ready!

View out the front.

Engine running and we're taxiing!

We sit close together.

Here's a short taxi video:

After a couple other departures, we were cleared to take off!

Takeoff video:

Check out that view!

Look, Ma. I'm flying the Trike!

And before I knew it we were in the air and flying.

The first few seconds were kid of freaky, given how open the cockpit is. But that quickly faded away and I began to really enjoy the flying. We headed a bit south and then over the first ridge toward the Lick Observatory on Mount Hamilton where we could play around a bit and I could get a better feel for flying the Trike.

Heading for the hills...

Me again.

All those houses...

Another great view.

Back there I got to try various flight maneuvers, really get a feel for the air, and even did some soaring in thermals. I was really impressed by how well we could feel the air as the wing tried to move and the distinct changes in temperature as we flew into different pockets.

Turns were easy once I got over the idea of forcing them. We also performed a few stalls (though they hardly compare to stalls in a larger airplane or glider)--very benign. I was impressed by how stable the aircraft was too. Let go and it'll pretty much fly straight and level if you're in calm air.

The view was, of course, amazing. And being able "feel" the air more directly makes you feel a lot more connected to the atmosphere you're flying in.

All in all it was a much more "raw" experience then the glider flying I'm used to. I really enjoyed it.

After almost an hour of flying, it was time to head back in and land.

Heading in to land.

Landing video:

After landing, we taxied back to the hangar so that Pete could pick up his next victim passenger.

Next up was Kathleen...

Strapped in.

Ready to go!

The took for for a similarly fun 1-hour flight.

Takeoff video:

I was there to grab a few pictures as the taxied back to the hangar area.

Thumbs Up!

What a blast!!!

Time to let someone else play...

Conclusion

We both had an absolute blast flying the trike. It's a little like flying a power plane and a little like flying a glider, but it's a very different experience from both. It's a very raw and natural form of flying that really puts you in touch with your surroundings and gives you an amazing view of the area you're flying in.

Thanks again to AeroDynamic Aviation for organizing this event. Not only do they provide excellent flight training and aircraft maintenance, Owner Zdravko Podolski has consistently organized events to help pilots expand our flying horizons.

We're both very tempted to get a weight-shift control add-on for our pilot certificates. It's worth noting that many Trikes make it easy to swap out the wheels on the landing gear for floats (land on lakes!) or skis (land on snow/ice).

Pictures and Video

I have the whole set in this Picasa album: Trike Flying at RHV

Or you can see a subset in my Trike Flying album on Flickr.

The videos are in my YouTube channel as well.

Loose Fuel Cap and Losing Fuel In-Flight (with Video)

While our flying trip to Yellowstone and the Grand Tetons was great, we had a few problems along the way. Like they say, bad things often come in groups of three. This was our third one. (The other two involved a hiding cat and a blown tail wheel on the airplane).

On our way home Saturday, we flew from Jackson, Wyoming (JAC) to Burley, Idaho (BYI) to get a bit of fuel, grab a snack, and take a restroom break. A few minutes in the air after leaving Burley, Kathleen looked back at the left wing and saw fuel streaming back. And it didn't seem to stop. Uh oh!

She shot this video.

We were closer to Twin Falls (TWF) at this point, so I called up the tower and began our descent down from 9,000 feet to land and check it out. We fully expected the fuel cap to be completely missing. Unlike the fuel caps on a Cessna, there is no chain that keeps it connected to the fuel tank. So if the cap comes off in flight (or was never put back on), you've lost it.

The tower cleared us for a straight-in on runway 25 and we landed shortly after a Turbo Ag Cat came in on the opposite runway.

We landed, taxied over to the full service fuel and FBO, and parked. I borrowed a a bucket (no ladder handy) from the guy who came over, climbed up, and was astonished to find the fuel cap was there and it felt tight!

There are two flanges on the fuel cap that serve to lock it to the neck of the fuel tank. Normally both of them are fitted tightly below the rim. But the guy who fueled it in Burley manage to put the cap on in such a way that only one of the two flanges was engaged below the rim.

When I pulled up on the back of the fuel cap, there was enough play to allow fuel to escape in flight when there's a substantial vacuum created by the airflow.

New lesson learned: Always pull up on the fuel cap to make sure it's on tight. Simply making sure that it won't turn anymore is not sufficient.

But that's not all!

With the situation apparently resolved, I got back in, started up the plane, and called up the ground controller. We were told to Taxi to runway 07 for a southbound departure. As we made our way over, I told Kathleen that I was expecting half the fuel we took on at Burley to be gone. But that was okay, since were could land and fuel up in Elko, Nevada (EKO) which was along our route of flight.

We got our takeoff clearance, took off, and departed the area to the south. About five miles from the airport, I glanced up at the fuel gauges and was shocked to see that the left tank was reading close to 1/4 and the right was reading 1/2. Normally I trust the right tank to be very accurate and doubt the left. But I haven't had enough time flying with it since the right tank was repaired, so I didn't really know how much fuel we had.

It didn't take more than a minute to decide that we had to land back at Twin Falls again and top off the tanks before continuing home.

Lesson #2 learned: Always check the fuel level after spewing gas in flight. It comes out a lot faster than you think (as confirmed by our mechanic later that evening).

In better news, we have a ton a great photos and video from our trip. Those should be coming on-line before too long.

Sunset over the Grand Tetons

Our trip to the Grand Tetons and Yellowstone is going very well. We've been pretty busy, so I haven't had a lot of time to write. But here's a picture of the sunset from tonight.

We're taking lots of pictures and video with the Flip and enjoying the scenery. Weather looks good for the next several days too.

Not a bad vacation at all. :-)