Jeremy Zawodny's blog

I'm Blogging Elsewhere

I haven't written much here recently. And that's really due to a lot of reason, not the last of which is being busy and that we have a wonderful new kitten named Bear:

The other issue is that I've becoming increasingly unhappy with my blogging tools, the custom hacks I've added over the years, and the required maintenance and upkeep.

So I've decided to give WordPress a try. You can visit my new experimental (likely semi-permanent) blog here: Jeremy Zawodny's WordPress blog

I may find myself posting stuff here from time to time. Maybe. But the reality is that I'll probably work on a mega export of this stuff into the the new blog and setup some redirects at some point.

When I have time.

Which means it may take a while. But that's life.

In the meantime, I have more efficient tools over there and am likely to publish more often. So check it out if you're interested. Or not.

Zip Line and Rappelling Adventure in Puerto Vallarta

A few weeks ago, Kathleen and I went on our first cruise together. It was her fifth and my first, so I got to experience a lot of new things. If time allows, I'll try to write about that in the future. But for now, I want to talk about one of our shore excursions.

Our first day in port was in Puerto Vallarta. The weather was absolutely perfect: a few scattered clouds, light winds, and about 80 degrees with bright sun--a far cry from the storms in California. We disembarked from the ship and met up with a group who were all going on the same excursion. We were then taken to what I can only describe as a "speed raft" designed to carry about 20-30 people. We boarded the boat and were treated to a high speed ride of roughly 20 minutes across the bay to a small beach and dock area.

The cruise across the bay gave us some view of very nice water-front houses. It appeared as though there are a lot of wealthy folks that decided to build vacation homes there.

At the dock across the bay, we got off the boat and were lead a few hundred feet to a guide that instructed us to board one of two 4x4 trucks outfitted with benches in the back. The trucks took us on a 15-20 minute ride through the town and into the forest and up to a higher elevation, ultimately arriving at the "base camp" for the operation.

There we were asked to leave behind anything we didn't want to get wet. That meant cameras, wallets, phones, etc. They repeatedly warned us that we'd be getting very wet. We were also told, much to our surprise, that we wouldn't need bug spray or sunscreen. Most of our time would be in the shade of trees.

After depositing our belongings to a communal bag that would be placed into a secured locker, guides helped get us suited up with our harnesses, helmets, gloves, and other gear. Then we were lead to small set of benches where the guides were introduced in a fairly comical fashion and they spent some time teaching us three important signals: speed up, slow down, pull your legs up. We were also instructed on proper hand placement on the zip line, posture (holding your legs up a bit), and braking techniques.

(I noticed during the introductions that there was roughly a 1:2 ratio of guides to participants.)

After that was out of the way, we headed over to stables where each of us was paired up with a mule for a ride up the path that leads to the top of the course.

The mule ride up was probably 15-20 minutes and ended near the base of the first rappelling platform. There we did some brief stretching, reviewed the signals, and had a chance to ask any last minute questions.

Then it was time to climb a short hill up to the first platform. This platform, unlike others we'd see later, was large enough to handle the entire group, so we could watch the procedure for getting hooked on to the lines. One of our guides went first so we could see what everything looked like--and that it was possible to arrive at the other end safely.

Ready to Go

After the first guide went across (he was on the other end to unhook us and catch us if we arrived a bit too quickly), the line of participants stepped up the the ropes, one at a time. One would get hooked up, checked, and then was told to go. While one was zipping, the next person would be hooked up and released once the first one was safely on the remote platform.

The first zip was reasonably long but a fairly shallow slope, so you couldn't get going too fast even if you didn't brake to slow down. I decided to see how fast I could get going before I felt like I should slow down. It turns out that it wasn't until I was approaching the platform and got the "slow down" signal that I needed any substantial braking. I did "tap the brakes" a couple times just to prevent myself form turning, since that's the only way to keep yourself facing forward as you go.

I arrived at the platform, stopped, and was unhooked.

"How was it?" asked the guide.

"Awesome!" I said.

I knew the rest of the day was going to be a lot of fun...

From there we went on to the second platform and were treated to a slightly faster (steeper) line and some more interesting scenery. Beyond that, I forget exactly how many more lines we did before coming to what the called the "assisted vertical zip", which I can only describe as a zip line on a 45 degree angle. It's "assisted" because they thread an additional rope through the harness is such a way that the guide down below (you drop down 80 feet or so) can adjust your descent rate.

Rappelling Fun

That got us to the point at which we could start our rappelling adventure. The guides swapped out a bit of gear on our harnesses and then lead us to the place from which we'd be rappelling--a waterfall.

There we two lines going down, so one person could be going down on one while the other was getting hooked up on the other. Both lines terminated in the pool of water at the bottom of the waterfall. The water appeared to be a few feet deep, but it was hard to say from the top.

They reviewed the techniques before letting us go down (of course) but before I knew it I was hooked up and "walking" down the rock face on the line nearest the waterfall. When I got to the bottom, I discovered that the water was about 2.5 - 3.0 feet deep--just enough to get my lets wet and touch the bottom of my shorts. I figured that's why they made the point about everyone getting wet--since they know we're going to "land" in the water and it may be high or lower depending on the waterfall. Plus you get wet from the waterfall itself if you went down on the line nearest to it.

It wasn't long before I discovered how wrong I was.

Splash Down!

One of the next zip lines we encountered was fairly short--maybe 150 feet long. But it started at the top of a set of rocks and carried you over those rocks (barely missing them) and then over another pool of water. However the zip line continued to descend until it reached the edge of the pool, which meant that using that zip line would get you REALLY wet.

I watched several folks ahead of me splash down into the water, some exclaiming because of the temperature (mountain water is always cold, right?). When it was my turn, I asked the guide if I needed to brake. He said it'd probably help but I didn't NEED to. So decided to go full speed down the line. Needless to say, I impacted the water and splashed down pretty well. Like most of the folks before me, I was SOAKED.

It was around then that I remembered that I had accidentally left my wallet in my pocket. Oh, well. I enjoyed the ride and the cold water actually felt refreshing to me.

Where is everyone?

After one of the next zips, Kathleen and I managed to get a bit behind and briefly separated from the group. But there was a path to follow, so we just went along expecting to meet up with everyone else.

Except that we didn't. Instead, we arrived at another platform with nobody there waiting for us. We saw some people in the distance (headed away) and called to them but they didn't hear or notice us. It was then that we realized we must have taken a wrong turn somewhere and were officially lost. In the Mexican forest.

We talked about backtracking or maybe crossing the water below us and continuing on that path, but we ultimately decided that it was probably best to sit tight and wait for the guides to notice we were missing and come find us. Surely this happened before and they know where to go looking, right?

Luckily that worked just fine. We didn't wait there more then 10 minutes before a pair of the guides appeared and asked what we were doing there--even joking about the two of us sneaking off for some "alone time" (if you know what I mean). They led us across the water on a rope and across another one before reaching the next spot where we met back up with the group.

Down we go!

Not long after that we came to the vertical descent. It was affectionately referred to later on by several folks as "walking the plank" because we'd walk out on a metal walkway, suspended about 70 or so feet above another pool of water, to reach a slight larger metal platform. There you'd be hooked up to a rope and instructed in how to use your had on the rope behind you to control your rate of descent. One of the guides down below also had a rope they could use to assist if you had trouble.

I'm not sure why this freaked some people out as much as it did. It was like a more aggressive form of the "assisted zip line" where you get more control. I had fun going down when it was my turn! (Even if my hand did slip a bit.)

We had a few more zips after that and before long it was time to hike back to the base camp. Our route back took us across numerous small streams. Often there were overhead ropes installed so you could hang on as you crossed in case you slipped on a rock or had balance problems. It was a nice way to wind down the day.

Finishing Up

Back at the base camp, we traded in our helmets, gloves, and harnesses for our stored possessions, washed our our shoes and had a chance to use the restroom. Then we had a bout 20 minutes to peruse the photos (see note below) and munch on some snacks. It was mid-afternoon and most of us hadn't eaten since breakfast, so the snacks were most appreciated! I especially enjoyed the chips and fresh salsa.

Hunger conquered and pictures purchased, it was time to hop back on the trucks for a ride down the hill and back to the boat that brought us across the bay. There were two very important differences about the ride down compared with the ride up. First, it was much faster. The diver took full advantage of gravity, so made good use of the seat belts and the overhead "oh, shit!" straps. It was quite a ride!

Mmm. Tequila!

The second difference is that we had a stop on the way back.

At a tequila factory.

For a tasting.

Yay, Mexico!

I really didn't know anything about the differences between types of tequila--that some are intended for use in drinks like margaritas while others are made for sipping (like a scotch, I guess). And I most definitely didn't know that flavored varieties like "almond" and "chocolate" existed!

I must say that the almond tequila was the best tequila I've ever had the pleasure of trying. And it's easily near the top of my list for "sipping" alcohols. The taste was just fantastic--very smooth. In retrospect, I should have bought a bottle or two to bring home. But at the time I figured they were a bit pricey and we might encounter other sources during our time in Mexico.

That may be my only regret about the entire trip.

Anyway, after the tequila tasting, we proceeded back down to the small beach and boat dock, hopped back on the "speed raft" and got a nice ride back to the ship.

Conclusion

All in all the excursion was a great time. We really enjoyed the zip line and rappelling experiences, all the modes of transit (boat, 4x4, mule), excellent weather, great scenery, and friendly people.

Note on Photos

Since we didn't take a camera along, I should note that all of the photos in this post and in this Flickr set were taken by the photographer "guide" who was on the course with us. She did a good job of capturing some of the scenery and nature along with each of us in varying states of descent on both zip lines and rappelling. We purchased a CD of the pictures at the end of the day. It contained all the photos of either one of us we found (starred them in Picasa) as well as all the scenery and nature shots she took that day. I was VERY COOL of them to give us the digital versions rather than trying to sell prints or something stupid like that. Thanks to Vallarta Adventures for being forward thinking.

Paper vs. Screen

I arrived at the Knoxville, TN airpot a bit ago for my slightly delayed flight back to San Jose via Dallas. Since I finished my book on the flight(s) out, I stopped by one of the shops to do something I rarely do: buy print media. Well, I still buy books now and then. But today I bought a couple of magazines: Disocver and Business Week.

I sat in the Ruby Tuesday's at the airport and managed to read a few articles in Business Week. After a bit I noticed that I was reading stories that I'd probably never read on-line. My on-line reading tends to be a lot more focused and directed. Reading for at a relaxed pace for pleasure is a whole different feeling.

It makes me wonder what I'd do with a Kindle or iPad if I had one. Would I use it the same way I read on a computer? Or would it be more like reading a magazine?

Hard to say. I have no plans to buy either device, but in a few years when I replace my existing portable computing devices, odds are that I'll have something tablet-like.

I'm not sure I'd want to be in the magazine business a few years from now. I feel like it's following the "lead" (if you can call it that) of the newspaper industry.

Interesting times.

Remembering Barnes

Yesterday was a very sad day for us.

Barnes, one of our four cats, had to be put to sleep after we took him to the pet hospital because he was having trouble breathing. A bit over a year ago, Barnes was diagnosed with Diabetes and he'd been getting regular insulin injections (2 per day) and vet visits. In fact, his most recent visit (3-4 weeks back) was very good. The doctor was happy with his progress and he generally seemed peppier. His blood test results were all very positive.

But yesterday when we noticed he seemed to be working harder to breathe, I knew something was very wrong.

He was placed into a 100% oxygen environment to make is breathing easier while they conducted some tests and gathered information. After some X-rays, we learned the extent of his troubles. He had an enlarged heart, fluid in his lungs, and cancer.

In a way, it's good we didn't see this coming.

His brother Noble seems to be taking it better than we are, and they'd been together their entire ~12 years. His newer brothers Timmy and Thunder will surely miss him as well.

His favorite activities in the world were eating chicken (and fish), exploring our catnip plants and toys, getting belly rubs, and napping on his favorite fuzzy blanket.

Barnes was the most loving and strong cat we'd ever known. He was probably hiding his illnesses for a long time, hoping for a few more belly rubs with us and visits with the catnip plants we grew.

We'll really, really miss him.

Here are a few of the pictures and videos we'll use to remember him...

Barnes as a Kitten

Barnes and Noble as Kittens

Banes on the Bed

Barnes Stretches

Barnes and Noble Cuddle

More Recent Barnes Photos

Barnes on the Floor

Barnes

Barnes and Noble

Barnes and Noble

Videos of Barnes

Barnes with a Catnip Toy

Our cats eating Chicken

Please remember to get your pets regular checkups with a good vet. They deserve it.

Sphinx and Gearman: A Distributed Computing AH-HA! Moment

A week ago I decided to finally get serious about putting gearman to use for search indexing. I had been batting the idea around in my head for a long time (too long, really) but figured I should just write the code and see what happens. It took less than a day to get a prototype working in our development environment, but the end result made me very happy.

Today, in our production deployment, when a sphinx cluster pulls new content to index, the master does all the work. It fetches the new and changed postings, massages them into the XML format that sphinx expects (and makes a lot of small changes along the way), invokes the indexer, and makes the new indexes available for the slaves. The second step is usually the most CPU intensive. Processing the raw data into XML involves a lot of other tweaks and changes that are very specific to Criagslist.

What I did was turn that into a gearman client/worker pair. The client (or master) simply submits processing tasks and then waits for each of them to complete. The workers fetch the data from the master, transform it, and make the transformed data available. When each task completes, the master grabs the transformed data an informs the worker that it can delete the file.

So instead of being stuck at using only the 4 CPU cores on a single box, I can run 4 workers on each of 3 machines and get 12 CPU cores involved. The end result is that I have a solid foundation for a system that can easily scale to many machines. AH-HA! Linear scaling rocks! So does relatively seamless distributed computing.

As time allows I'll have to work on deploying this in production.

On the MyBlogLog Shutdown

Marshall Kirkpatrick is reporting that Yahoo! will shut down MyBlogLog next year. Well, color me unsurprised. The service has languished for years. I removed it from my site a long time ago. It made me a little sad to do so, but it was just slowing things down and not really "adding value" as they like to say.

It's sad because I was involved in the MyBlogLog acquisition at Yahoo! and believed in what they were doing. I worked to help get the team on board, nag the right people to make sure they got reasonable hardware on which they could grow, interviewed their first post-Yahoo engineer, and made the trek up to the Berkeley office a few times a week to help them transition into Yahoo and work on plans.

I genuinely had high hopes for what MyBlogLog could do both inside and outside of Yahoo. But as I wrote in Watching Yahoo's Transformation:

MyBlogLog has all but died on the vine, right? Is there anyone left of the original team of 5 or 6 engineers still working on it? No, I think it fell victim to Yahoo's larger social strategy. FAIL.

On the one hand, it's sad that our collective time was wasted, but the members of the MyBlogLog team have all gone on to bigger and better things outside of Yahoo. And I suspect everyone involved learned some important lessons along the way.

Trust Oracle? Why?

In a 10-point press release issued today Oracle has listed a series of "commitments" regarding their acquisition of MySQL by way of acquiring Sun.

I am not impressed.

As a former employee of a large Internet company (the largest at the time, in fact) that used both Oracle and MySQL, I'm utterly puzzled by this. I can't think of why we should trust Oracle to do right by the users of MySQL--especially the non-paying users.

You see, for years Oracle worked agressively behind the scenes to discredit MySQL and tried hard to understand how their customers could ever consider using such a "toy" instead of their flagship product. In fact, it was so important to Oracle that they offered some very substantial discounts to customers who were using MySQL and Oracle. In some cases the discounts were so impressive that their motivation was clear: cut off the opportunity for MySQL to grow and spread in such organizations. (Remember what happened to Netscape when Microsoft gave away Internet Explorer for free?)

The funny thing is that it really didn't work. MySQL was already a fast moving train with lots of momentum. And it was still accelerating.

It was clear that Oracle saw MySQL as a threat to their business. When they eventually bought Innobase (the company that makes the InnoDB storage engine), many of us got more than a bit nervious. That put Oracle in a position of having a choke hold on the one componenet that was critical to MySQL's future success. They could have just shut down development entirely. But that may have made their motives a bit too clear.

Since then they've continued to develop InnoDB. However, the pace hasn't exactly been agressive and their openness around that has left me (and others) wondering what their longer term plans really are. The few tidbits we get seem to be overly vague. Could they have been throttling development of InnoDB? Or not providing the same resources that MySQL (and now Sun) would have? It's hard to say.

But here's the thing that continues to bug me...

Back a few years ago when Oracle dismissing MySQL in public while working hard against it in private, I realized that they were simply trying everything they could to protect their crowned jewels: public denials and classic FUD paired with hush-hugh backroom deals.

Nobody has managed to explain, in even a mildly convincing way, what has changed since then. Why should we suddenly trust Oracle? Their crowned jewels are still threatened by MySQL.

Convince me.

See Also: Monty's appeal is selfless!

Weird Email People Send Me

I really have no idea what prompted this:

This is Rev.Willson and am interested in some of your Barrel. I need the Barrel in the specifications of 55 Gallon Barrel. Blue Plastic.So what will the total cost of the Barrel with the dimensions i gave and i need 100 quantities of the Barrel.So i will like you to go ahead and quote me the total charges of 100 quantities of the Barrel + tax without shipping charges included.Because i will be picking this up as soon as it is ready from your location.Please if you don't have the type of barrel that am interesting kindly email me back with the types that you have.And i will be very glad if you can also email me back with the types of credit card that you will accept for payment. Please advice. Thank You.

It's so random that it's funny.

Speaking at San Francisco MySQL Meetup

I'll be speaking Tuesday night at the San Francisco MySQL Meetup. Here's the blurb from the announcement:

Jeremy Zawodny will talk about Craigslist's current use of MySQL, where it's painful, and how things are being re-architected to make the pains go away. This includes hardware changes, sphinx, redis, memcached, and some custom Perl work.

Despite not living too far away from the SF MySQL Meetup venue, I'm a little annoyed at myself for never having attended. So this will be my first time and I'm really looking forward to meeting local like-minded folks.

Thanks to Mike and Erin for recruiting me.

If you're coming and have specific things you'd like me to talk about, drop me a note in the comments or on Twitter.

Fixing Poor MySQL Default Configuration Values

I've recently been accumulating some MySQL configuration variables that have defaults which have proven to be problematic in a high-volume production environment. The thing they all have in common is a network blip or two can trigger some very undesirable behavior.

max_connect_errors

If a client is having trouble connecting to MySQL, the server will give up waiting after connect_timeout seconds and increment the counter which tracks the number of connect errors it has seen for the host. Then, when that value reaches max_connect_errors, the client will be locked out until you issue a FLUSH HOSTS command. Worse yet, if you have occasionally network blips and never need to restart your MySQL boxes, these errors can accumulate over time and eventually cause you middle of the night pain.

See Host 'host name' is blocked in the MySQL docs. Sadly, there is no way to disable this check entirely. Setting the variable to 0 doesn't accomplish that. Your only real solutions are (a) setting it to a very high value (max_connect_errors=1844674407370954751), and (b) running an occasional FLUSH HOSTS command.

connect_timeout

This is related to the above problem. In situations of network congestion (either at the client or server), it's possible for an initial connection to take several seconds to complete. But the default value for connect_timeout is 5 seconds. When you trip over that, the max_connect_errors problem above kicks in.

To avert this, try setting connect_timeout to a value more like 15 or 20. And also consider making thread_cache_size a non-zero value. That will help in situations when the server occasionally gets a high number of new connections in a very short period of time.

skip-name-resolve

MySQL does a reverse DNS lookup on every incoming connection by default. This sucks. It seems that no matter how good your infrastructure is, there are blips in DNS service. MySQL's host cache exists to keep those lookups to a minimum. Yet I've seen this cause pain off and on for eight years now. I can only assume there's a bug in the host cache or the resolver library when this happens.

I recommend adding skip-name-resolve to your /etc/my.cnf to skip DNS entirely. Just use IP addresses or ranges for your GRANTs. It seems that slow replies from DNS servers can also help you to trip over connect_timeout as well. Imagine having 2 or 3 DNS servers configured but the first one is unavailable.

slave_net_timeout

When the network connection between a master and slave database is interrupted in a way that neither side can detect (like a firewall or routing change), you must wait until slave_net_timeout seconds have passed before the salve realizes that something is wrong. It'll then try to reconnect to the master and pick up where it left off. That's awesome.

However, the default value is 3600 seconds. That's a full hour! FAIL.

Who wants their slaves to sit idle for that long before checking to see if something might be wrong? I can't think of anyone who wants that.

My suggestion, if you're in a busy environment, is that you set that to something closer to 30 seconds.

Recent Sphinx Updates

If you use the Sphinx search engine and have been watching the development branch (0.9.10) and wondering when to upgrade, I'm here to tell you that "now" is a great time. As of r2037, the last major issue I regularly saw has been fixed. The other big bug was fixed in r2031.

Late last week I began testing those fixes in a "burn-in" test I've developed that makes liberal use of indextool --check. Instead of seeing index corruption within an hour, I saw none. After 3 days of no failures, I deployed it to a subset of our search back-end servers. Yesterday we deployed it to half of the remaining servers.

So far, so good!

I should note that all our index corruption was merge related. Sphinx wasn't building corrupt indexes out of the box, but the merges (usually filtering merges) could produce corrupted indexes.

We were upgrading from a lightly patched version of r1894. That meant rebuilding our indexes to use the new and more compact format. Some of the obvious benefits of the upgrade:

• smaller disk and memory footprint
• pre-fork support to spawn searchd children at start up
• more reliable shutdown and pid file handling
• kill lists
• mysql protocol support
• lots of small optimizations and fixes

Thanks to the Sphinx team for their excellent work. I look forward to the release of Sphinx 1.0.

Oracle and MySQL

In The EU and MySQL, Tim Bray treads lightly on the topic of Oracle's pending ownership of MySQL if the Sun acquisition goes through. I left a comment on his post, but he's likely to be heavily moderating what appears there since he works for Sun.

So here's what I posted on his blog.

I haven't yet seen anyone explain what motivation Oracle has for pouring resources into MySQL, especially if it eats away at their DBMS business on the low end.

I've been puzzling over this since their acquisition of Innobase Oy (the makers of InnoDB) years back. Is Oracle serious about seeing MySQL grow and succeed, or was that just a way to get a strangle-hold on a critical piece of MySQL?

I've never had the chance to ask Ken Jacobs that. Actually, I have but it would have been kind of rude. And even if I did, I'm not sure I could trust the answer.

I doubt this comment will get published, but as a MySQL long time user, supporter, advocate, and author I'm really glad to see things like PBXT, MariaDB, and Percona's XtraDB out there.

Really, we need that kind of diversity in Open Source. A MySQL/InnoDB "monopoly" wouldn't have been healthy in the long run.

A reporter contacted me today to ask, among other things, if I think Oracle was/is threatened by MySQL. Oracle claims that they serve two different markets, etc. He wasn't so sure.

Sadly, there's some background information that I should not publish here, but suffice it to say that Oracle was and probably still is threatened by MySQL. Their sales/marketing tactics made this quite clear long ago. But those deals were rarely public--for good reason.