Jeremy Zawodny's blog

Speaking at San Francisco MySQL Meetup

I'll be speaking Tuesday night at the San Francisco MySQL Meetup. Here's the blurb from the announcement:

Jeremy Zawodny will talk about Craigslist's current use of MySQL, where it's painful, and how things are being re-architected to make the pains go away. This includes hardware changes, sphinx, redis, memcached, and some custom Perl work.

Despite not living too far away from the SF MySQL Meetup venue, I'm a little annoyed at myself for never having attended. So this will be my first time and I'm really looking forward to meeting local like-minded folks.

Thanks to Mike and Erin for recruiting me.

If you're coming and have specific things you'd like me to talk about, drop me a note in the comments or on Twitter.

Fixing Poor MySQL Default Configuration Values

I've recently been accumulating some MySQL configuration variables that have defaults which have proven to be problematic in a high-volume production environment. The thing they all have in common is a network blip or two can trigger some very undesirable behavior.

max_connect_errors

If a client is having trouble connecting to MySQL, the server will give up waiting after connect_timeout seconds and increment the counter which tracks the number of connect errors it has seen for the host. Then, when that value reaches max_connect_errors, the client will be locked out until you issue a FLUSH HOSTS command. Worse yet, if you have occasionally network blips and never need to restart your MySQL boxes, these errors can accumulate over time and eventually cause you middle of the night pain.

See Host 'host name' is blocked in the MySQL docs. Sadly, there is no way to disable this check entirely. Setting the variable to 0 doesn't accomplish that. Your only real solutions are (a) setting it to a very high value (max_connect_errors=1844674407370954751), and (b) running an occasional FLUSH HOSTS command.

connect_timeout

This is related to the above problem. In situations of network congestion (either at the client or server), it's possible for an initial connection to take several seconds to complete. But the default value for connect_timeout is 5 seconds. When you trip over that, the max_connect_errors problem above kicks in.

To avert this, try setting connect_timeout to a value more like 15 or 20. And also consider making thread_cache_size a non-zero value. That will help in situations when the server occasionally gets a high number of new connections in a very short period of time.

skip-name-resolve

MySQL does a reverse DNS lookup on every incoming connection by default. This sucks. It seems that no matter how good your infrastructure is, there are blips in DNS service. MySQL's host cache exists to keep those lookups to a minimum. Yet I've seen this cause pain off and on for eight years now. I can only assume there's a bug in the host cache or the resolver library when this happens.

I recommend adding skip-name-resolve to your /etc/my.cnf to skip DNS entirely. Just use IP addresses or ranges for your GRANTs. It seems that slow replies from DNS servers can also help you to trip over connect_timeout as well. Imagine having 2 or 3 DNS servers configured but the first one is unavailable.

slave_net_timeout

When the network connection between a master and slave database is interrupted in a way that neither side can detect (like a firewall or routing change), you must wait until slave_net_timeout seconds have passed before the salve realizes that something is wrong. It'll then try to reconnect to the master and pick up where it left off. That's awesome.

However, the default value is 3600 seconds. That's a full hour! FAIL.

Who wants their slaves to sit idle for that long before checking to see if something might be wrong? I can't think of anyone who wants that.

My suggestion, if you're in a busy environment, is that you set that to something closer to 30 seconds.

Recent Sphinx Updates

If you use the Sphinx search engine and have been watching the development branch (0.9.10) and wondering when to upgrade, I'm here to tell you that "now" is a great time. As of r2037, the last major issue I regularly saw has been fixed. The other big bug was fixed in r2031.

Late last week I began testing those fixes in a "burn-in" test I've developed that makes liberal use of indextool --check. Instead of seeing index corruption within an hour, I saw none. After 3 days of no failures, I deployed it to a subset of our search back-end servers. Yesterday we deployed it to half of the remaining servers.

So far, so good!

I should note that all our index corruption was merge related. Sphinx wasn't building corrupt indexes out of the box, but the merges (usually filtering merges) could produce corrupted indexes.

We were upgrading from a lightly patched version of r1894. That meant rebuilding our indexes to use the new and more compact format. Some of the obvious benefits of the upgrade:

• smaller disk and memory footprint
• pre-fork support to spawn searchd children at start up
• more reliable shutdown and pid file handling
• kill lists
• mysql protocol support
• lots of small optimizations and fixes

Thanks to the Sphinx team for their excellent work. I look forward to the release of Sphinx 1.0.

Oracle and MySQL

In The EU and MySQL, Tim Bray treads lightly on the topic of Oracle's pending ownership of MySQL if the Sun acquisition goes through. I left a comment on his post, but he's likely to be heavily moderating what appears there since he works for Sun.

So here's what I posted on his blog.

I haven't yet seen anyone explain what motivation Oracle has for pouring resources into MySQL, especially if it eats away at their DBMS business on the low end.

I've been puzzling over this since their acquisition of Innobase Oy (the makers of InnoDB) years back. Is Oracle serious about seeing MySQL grow and succeed, or was that just a way to get a strangle-hold on a critical piece of MySQL?

I've never had the chance to ask Ken Jacobs that. Actually, I have but it would have been kind of rude. And even if I did, I'm not sure I could trust the answer.

I doubt this comment will get published, but as a MySQL long time user, supporter, advocate, and author I'm really glad to see things like PBXT, MariaDB, and Percona's XtraDB out there.

Really, we need that kind of diversity in Open Source. A MySQL/InnoDB "monopoly" wouldn't have been healthy in the long run.

A reporter contacted me today to ask, among other things, if I think Oracle was/is threatened by MySQL. Oracle claims that they serve two different markets, etc. He wasn't so sure.

Sadly, there's some background information that I should not publish here, but suffice it to say that Oracle was and probably still is threatened by MySQL. Their sales/marketing tactics made this quite clear long ago. But those deals were rarely public--for good reason.

Lazy Shopping Carts

What's with all the lazy shopping carts? It seems that through the years, shopping carts have become progressively more lazy.

Years ago, when a shopping cart was done with its duty, it had to find its way all the way back from your car to the store entrance--either just inside or just outside. But, apparently as the result of an aging population of shopping carts, grocery store designers started scattering "cart corrals" around the parking lot so that the poor carts only had to scoot a few dozen yards before coming to rest. But even that is starting to seem inadequate.

I was at Trader Joe's earlier today and noticed quite a number of carts that could only make it as far as the little curbs that surround the trees that in the parking lot. And at Safeway, where there seem to be cart corrals every 15 parking spots, I still see carts that aren't able to make it even that far.

Modern technology has done so much to our modern food system in the last 20 years (a fair amount of it quite questionable, I'm sure), yet so little has been done to help these carts! Why is that?

Who are the cruel people that insist on keeping these carts in service long after they're able to perform up to very basic expectations?

What is this world coming to?

It's sad, really.

Pass the Duct Tape

Reading The Duct Tape Programmer really hit a nerve with me. If you code for a living (or part of your living), go read it. I sometimes get annoyed by Joel Spolsky's writing, but this time he hit the nail right on the head.

I am by no means putting myself in the same league as Jamie Zawinski, but I definitely subscribe to the idea that keeping my code as simple as possible is going to pay big, big dividends in the long run.

Needless complexity really does kill you.

In my work it has manifested itself in a few ways. But one that seems to pop up over and over in recent times (and has bitten a few of my coworkers as well) is socket programming. For whatever reason, we're able to find the the freely available client libraries for interacting with some services will inevitably fail a small but non-zero percentage of the time. And when you're making millions and millions of calls or connections per day, even a 0.01% failure rate is enought to make for a really bad day.

That's especially true when the failure mode involves leaking file descriptors or needlessly long timeouts (or timeouts that fail to work properly) that result in a full or partial cascading failure. It's the ultimate in frustration.

I'm not going to name names here. Maybe at an upcoming conference I'll talk about what we've seen and how network oddities add a whole new set of failures to the mix.

Anyway, the solution ends up being one of two options: (1) forking the module and re-writing the code to use the low-level system calls instead of the multitude of abstraction layers that were supposed to make the task easier (die abstraction layer! die!), or (2) scraping the module entirely and writing your own. The first option sucks but at least you're not having to learn all the quirks of those abstraction layers. The second option sucks even more but at least you know exactly what you've got when it's all said and done.

For me it's more satisfying and productive get to know the low-level stuff well enough that you can just drop it in when you need it. It's less frustrating than trying to reproduce all the conditions and variable that cause some abstraction layer to fail.

Pass the duct tape. I've got work to do!

Watching Yahoo's Transformation

Having not worked at Yahoo in a bit over a year, it's interesting, amusing, and frustrating to watch what's going on over there. Some of their moves confirm feelings I've had for some time, while others are more puzzling.

The Search deal with Microsoft was basically inevitable. You could see even two years ago how out-gunned Yahoo was compared to Google and to what Microsoft was rumored to be thinking of doing. Based on what I knew, Yahoo was spending less than 20% of the money on search that Google was and they were trying to do the same worth with about 1/10th the hardware and less mature infrastructure software. Meanwhile, the web kept growing and became more and more real-time.

It seems that Yahoo wants to sell Zimbra. I remember when that acquisition happened. I thought it was cool technology and would be great if we actually wanted to compete in the on-line "office suite" market (or whatever), but that never happened. Instead it remained as one of those "enterprise" products that Yahoo has a history of trying and failing at. Remember Yahoo's enterprise instant messaging product? FAIL.

Now comes news that Yahoo wants to sell Small Business which includes their web hosting and domain registration businesses. This makes good sense to me. While it's a business that I think has usually made money, it simply wasn't competitive in a day and age when you can get a full virtual machine, storage, and bandwidth from any number of vendors who aren't scared to offer good remote access (ssh anyone?) to the server: Slicehost, Rackspace, Server Beach, Amazon, and so many others. Yahoo's offering may have made sense back in the year 2000 or so when it really competed on price, but this is one of those race-to-the-bottom commodity business and has been for years.

People picking on Carol for selling some stock recently? Well that's just dumb. Yahoo is a public company and she really didn't sell that much.

I hope that Carol is able to trim the parts of Yahoo that no longer make sense and help bring some focus to the company. I really do. But to be honest, I've seen it before. Terry Semel tried to do something similar when he came on board. But that was a wandering effort that ultimate lacked focus and wasn't ambitious or forward-thinking enough.

Jerry and Sue tried this when Terry left, but I really think they were too "Yahoo" to transform Yahoo into what it needs to be.

I see how Carol is trying to be smart on the business side, but I'm not sure how Yahoo plans to wow its users. And coming from the Yahoo! Developer Network, I wonder if Yahoo will ever get serious about outside developers. Conferences and Hack Days are great, but I suspect they still haven't figured out how to offer buisness-class APIs (with an exchange of money and an SLA). Aside from YUI and Hadoop, can you really go beyond a prototype with this stuff?

It could simply be that I never really drank the YOS/YAP kool-aid and never will get it.

Oh, and what about those smaller startups? Should I start to worry that del.icio.us or Flickr is going to go away? What about Upcoming? When their founders start to worry, I feel like I should too. At least Flickr has a business model and appears to still be kicking some ass. MyBlogLog has all but died on the vine, right? Is there anyone left of the original team of 5 or 6 engineers still working on it? No, I think it fell victim to Yahoo's larger social strategy. FAIL.

I hope that Carol can be clear, focused, and agressive in re-shaping Yahoo. The half-measures attempted over the last few years simply haven't been enough and never will be.

How my netbook taught me to love xmonad

I've had this low-level urge to try a new window manager for a few months now. I work on a Linux box (Ubuntu) daily and mostly run a number of terminals, GNU Emacs, Mozilla Firefox, and Google Chrome. Nothing too fancy, really. Oh, and a shitty VPN client.

Background

Most of the time I'm doing this in front of a 24" (or larger) monitor running at 1920x1200, so there's a lot of screen real estate. Yet I was always annoyed by how much time I spent moving windows around or trying to find the optimal layout--always reaching for the mouse.

Years ago when 1024x768 was the norm, I ran a heavily customized fvwm2 and enjoyed it. But then I made the move to Windows for a few years and came back to Linux with Ubuntu/Gnome as my "desktop."

The Netbook

Months ago I wrote about how I love my Samsung NC10. When I'm not at my desk, I'll often use PuTTY to login, resume my screen session(s), and continue working. For what it's worth, I find that the free DejaVu Fonts (specifically the Monospaced one) works exceptionally well.

What I realized is that my method of having one terminal in full-screen mode on each "desktop" (thanks to the VirtualWin virtual desktop manager) is surprisingly productive, even on the little 10" screen. At first I considered this a fluke and attributed it mainly to the novelty of working this way. But after a while I realized that it was the focus that this setup enforces. There simply isn't enough room to have a browser on screen to distract me while I'm coding something, reading email, etc.

I really need to focus one or a few tightly realted tasks. The cognitive overload of having the whole Internet available really gets pushed off-screen and mostly out of mind.

Trying xmonad

After a discussion in our chatroom at work the other day, I finally decided to give a new window manger a try: xmonad. A big help was Tom's Introduction to the xmonad Tiling Window Manager which gave me just the information I needed to get started.

I used it most of Friday and a bit off and on Saturday, both on my primary work computer and my "home" Linux desktop machine. The experience has been surprisingly positive so far. Most of the hassles have revolved around re-training my hands to learn some new keyboard shortcuts and finding replacements for the few GUI things that Gnome provided on my previous desktop.

On thing I particularly like is that most of the keybindings seem very sane out of the box with xmonad. I haven't really needed to customize anything yet. I have found that a couple keystrokes that I use in GNU Emacs appear to be intercepted by xmonad and I haven't found an easy way to undo that or at least discover what they're supposed to do: Alt-w and Alt-q are the two I've noticed.

I also needed to resurrect an old xmodmap file that I could use to turn my CAPS LOCK into a Control key and re-discover the right xset command to set my key repeat rate higher than the default: xset r rate 250 30.

Other than those few nits, it's been pretty smooth sailing. I definitely feel like I'll be more productive in the long run a result of switching.

Have you tried a tiling window manager? Did you stick with it?

Pandora vs. CDs

I've been using Pandora for streaming music on-line for a while now. I enjoy it so much that I pay the roughly $5-$6 per month for a higher quality stream and because I want to do my part to make sure they stay viable as a business.

What's so great about it?

Pandora is unobtrusive, easy to use, and really good at finding similar and related music that I'm likely to enjoy. I'm always amazed at how good it is when I create a new station.

Just a few weeks back, I told Pandora to create a "U2" station. A few hours later, I realized that I'd been listening to an excellent selection of tunes.

I never get bored of the music on Pandora. It's just a part of my day, always playing good music to keep me happy while I'm coding or fixing things.

I feel like an idiot for the thousands of dollars I spent on CDs years ago, not to mention all the time and effort that went into digitizing that library multiple times (my own code, iTunes, WinAmp, etc.).

If I thought I could get any real money for my 550 CDs, I'd just sell 'em all and send the Pandora crew a nice check with a note: "Keep up the great work."

If you've not yet used Pandora, I highly recommend it. You'll be surprised by how often it plays music you like, even with the most minimal input from you.

Firefox vs. Google Chrome Revisited

Last week, in Google Chrome is the New Firefox, and Firefox the new IE, I ranted a bit about how slow Firefox 3 (notably tab switching and the "awesome" bar") was on my Ubuntu 9.04 machine. Needless to say, I got some good feedback from that post and it prompted it me to do a few things.

I've since been running both Firefox 3.5 as well as the Firefox 3.6 trunk code and can say that both are notably faster than Firefox 3.0.xx. The difference between 3.0.xx and 3.5 was substantial and really helped to close the gap with Chrome. Going to the 3.6 alpha nightly builds made it even faster in some places and slower in others--not surprising since it's still in development. Scrolling was drastically worse, but I'm told that's currently in flux.

I have to hand it to the Firefox team. They're not taking Chrome lying down. I've found no real issues with running 3.5 so far and it's a bit of mystery to me (which is to say "I haven't researched at all...") why Ubuntu 9.04 isn't upgrading folks to it. I'm really looking forward to seeing 3.6 stabilize. The Javascript and layout performance seems really good in my use so far.

Google Chrome is the New Firefox, and Firefox the new IE

I spent too long on Friday screwing around with stuff on my work laptop in an effort to make Firefox's apparent performance not SUCK ASS. Ever since I upgraded to Ubuntu 9.04 I've been somewhat unhappy, mostly as a result of the well publicized issues with Intel Video on Ubuntu 9.04.

I read about possible hope with upgrading the driver which also required a kernel upgrade, so I did both and rebooted. And, as I hoped, video seemed a bit snappier.

But Firefox still SUCKED ASS.

At this point I was REALLY PISSED. Sure my new video was nice and all but making new tabs (or switching between them) was still slow, and the disaster known as the "awesome bar" (how to disable) still sucked.

So on a whim I went and installed Google Chrome. It totally rocks on my Samsung NC10 netbook (running WinXP), so I figured what not give it a try.

It turns out that Chrome on Linux is DRAMATICALLY FASTER THAN FIREFOX!.

It's been quite stable on Windows, so I'm hoping the same is true on Linux and I can just switch over to it. As of now, Firefox is my primary browser on only half my computers. Chrome seems to be slowly displacing it, just like Firefox replaced the bloated pig known as Mozilla years ago (and the long since stagnant IE on Windows).

It's funny. Browsers seem to be like Internet companies. Every few years a new, small, faster one comes along to kill off some (or all) of the previous generation. I guess this is just the latest in that constant evolution.

It'll be interesting to see how this new competition really affects Mozilla Firefox.

I spend most of my day in gnome-terminal (to screen, mutt, irssi, etc.), GNU Emacs, and a browser. When they're not fast and stable, my life sucks.

Flying an Ultralight Trike or Weight Shift Control Aircraft

On Saturday Kathleen and I took advantage of the chance to take a short class in Trike flying and then go up for introductory rides.

This was an event organized by AreoDynamic Aviation (formerly Amelia Reid Aviation) at the Reid-Hillview Airport in San Jose.

We arrived to find Pete Marsh's Antares MA-34 R912 Ranger with an 80 horsepower Rotax 912 engine sitting outside the hangar. So I took several photos of the engine, cockpit, and wing while we got a good look at it.

A view of the cockpit.

Beefy landing gear!

The Rotax 912 Engine

Glass Cockpit

After checking it out, Pete spent about an hour telling our class about Trike flying in Alaska, answering our questions about the sport and the machines, and giving us a little bit of an idea what to expect.

For the record, this Trike has electric trim and will cruise at about 70-75mph while burning 2.5 gallons of high octane auto gas per hour. So mileage is comparable to most cars. But the ride is a hell of a lot more fun! And with a 15 gallon fuel supply, you can fly a lot longer than your bladder will let you.

Several of us had reserved times to fly 1 hour $99 introductory rides with Pete, and before I knew it my time had arrived. I was really looking forward to the experience.

Pete helped me get strapped in and I took a few pictures and a vide as we taxied over to the run-up area for runway 31R.

My turn!

Strapped in and ready!

View out the front.

Engine running and we're taxiing!

We sit close together.

Here's a short taxi video:

After a couple other departures, we were cleared to take off!

Takeoff video:

Check out that view!

Look, Ma. I'm flying the Trike!

And before I knew it we were in the air and flying.

The first few seconds were kid of freaky, given how open the cockpit is. But that quickly faded away and I began to really enjoy the flying. We headed a bit south and then over the first ridge toward the Lick Observatory on Mount Hamilton where we could play around a bit and I could get a better feel for flying the Trike.

Heading for the hills...

Me again.

All those houses...

Another great view.

Back there I got to try various flight maneuvers, really get a feel for the air, and even did some soaring in thermals. I was really impressed by how well we could feel the air as the wing tried to move and the distinct changes in temperature as we flew into different pockets.

Turns were easy once I got over the idea of forcing them. We also performed a few stalls (though they hardly compare to stalls in a larger airplane or glider)--very benign. I was impressed by how stable the aircraft was too. Let go and it'll pretty much fly straight and level if you're in calm air.

The view was, of course, amazing. And being able "feel" the air more directly makes you feel a lot more connected to the atmosphere you're flying in.

All in all it was a much more "raw" experience then the glider flying I'm used to. I really enjoyed it.

After almost an hour of flying, it was time to head back in and land.

Heading in to land.

Landing video:

After landing, we taxied back to the hangar so that Pete could pick up his next victim passenger.

Next up was Kathleen...

Strapped in.

Ready to go!

The took for for a similarly fun 1-hour flight.

Takeoff video:

I was there to grab a few pictures as the taxied back to the hangar area.

Thumbs Up!

What a blast!!!

Time to let someone else play...

Conclusion

We both had an absolute blast flying the trike. It's a little like flying a power plane and a little like flying a glider, but it's a very different experience from both. It's a very raw and natural form of flying that really puts you in touch with your surroundings and gives you an amazing view of the area you're flying in.

Thanks again to AeroDynamic Aviation for organizing this event. Not only do they provide excellent flight training and aircraft maintenance, Owner Zdravko Podolski has consistently organized events to help pilots expand our flying horizons.

We're both very tempted to get a weight-shift control add-on for our pilot certificates. It's worth noting that many Trikes make it easy to swap out the wheels on the landing gear for floats (land on lakes!) or skis (land on snow/ice).

Pictures and Video

I have the whole set in this Picasa album: Trike Flying at RHV

Or you can see a subset in my Trike Flying album on Flickr.

The videos are in my YouTube channel as well.