I've always love reading jwz stuff, and it's not because we both have the same first and last initials.
I think I still enjoy writing software, usually. But what I end up spending almost all of my time doing is sysadmin crap. I hate it. I have always hated it. Always. If you made a Venn diagram, there would be two non-overlapping circles, one of which was labeled, "Times when I am truly happy" and the other of which was labeled, "Times when I am logged in as root, holding a cable, or have the case open."
Heh.
The defective yeti isn't fond of Culture Jam. I read it a year or so ago and joined Adbusters as a result. I liked the book, but more importantly, I liked the message. It resonated with me. A lot. Yes, there were things about the book that bugged me, but I overlooked that in light of the more important messages it contained.
If you haven't read Culture Jam, please do. It's worth reading even if you end up disagreeing with everything it says. You should also read Fast Food Nation as well.
One of these days, I'm gonna put a books category on my blog. But today is not that day. Neither will tomorrow be. Another book looms on the horizon. In fact, I need to update the crappy ass poor excuse for a web site I've got there. Perhaps I should make it more weblog-like. Maybe I'll use MovableType and spruce it up.
Yeah. Anyone want an RSS feed for progress updates on my book? Well, anyone other than my editor, that is?
Oh, you might be asking yourself why I work for a big media company if I'm a culture jamming fan. Well, I'll answer that later.
Is it just me, or is RSS really, really close to critical mass. You, know the kind where you read article in The Wall Street Journal about it. Well, Dave notes that "I just learned that RSS is being taught at a computer class at Harvard." and goes on to note that there are two article in the San Jose Mercury News: this one talk about the major RSS aggregators--both web sites and desktop/server software, and this one even mentions our little Finance News RSS test.
That second article has some other interesting tidbits.
One RSS developer said most major media companies worry about news feeds driving people away from their advertising-supported Web sites.
Uhm... DUH! But if these sites still haven't learned the "advertising lesson" yet, when will they? Ever?
Nonetheless, more mainstream sites are coming around. The BBC announced in August the availability of four news feeds. And the New York Times recently started making feeds available users of Radio UserLand software.
I have to say that the BBC rocks. They've been doing great work in streaming audio--even providing experimental Vorbis feeds. I tried those out a while back and rather impressed. Generally, the BBC seems technologically open-minded and likely to "do the right thing" or something close to it.
Here's my favorite part:
As for the future of RSS, developers are discovering new uses for the technology seemingly everyday. Yahoo is beta-testing a financial news feed. Web enthusiasts have recently figured out how to suck Amazon book titles and Google news headlines into news readers. And other developers are fiddling with ways to import calendar and weather content.
No, not because it mentions one of my projects. I've been named in the press enough recently. I like that because it's very clear about the future. The handwriting is on the wall: Provide RSS or someone will do it for you!
I hope the right folks are listening. No, not to me. Why would they. But listening to the RSS buzz. It's getting louder and louder. You can ignore it, but it is NOT going away. There's a good reason that NetNewsWire is one of the hottest applications to hit the OS X desktop this year. Part of that is because Brent rocks, but a lot of it part of a much larger shift that's taking place. And with all such shifts, you either get with the program or find yourself wondering how and why you missed it.
Shoot, Josh even added TrackBack and RSS (though I don't think he's announced that part yet) to Entropy, the system that powers Fruhead.com.
If you're still reading, I highly recommend reading Clay Shirky's Weblogs and the Mass Amateurization of Publishing. He has ideas that you probably haven't thought about yet.
Last week, I noticed that I was on Dave's radar. Now that I've caught up on blogging, it seems that I'm on Windley's radar too. He picked up on my FreeBSD/Linux MySQL story and that provoked an interesting discussion of "Abundance Mentality" that has continued here.
Specifically, Phil says:
Blogging requires what is called an "abundance mentality." If you don't approach it with the mental attitude that there's plenty to go around, you're less likely to share, which is at the heart of blogging. The cynical side of me wonders if this might not be blogging's fatal flaw: it requires a fair amount of altruistic behavior.
I have to admit that I never really thought about it before. But now that I do, there are parallels between blogging and Open Source.
However, when you look at the larger blogging sphere, like all the younger kids who are getting into it, things are a little less clear. But in the realm of sharing professional information, I agree so far.
As noted in the inluminent weblog there don't seem to be many bloggers at OSXCon. At least they're not blogging. That's rather disappointing. They were strong at OSCON and the Emerging Technologies Conference (damn, I wish I had the foresight to go to that one).
Check out the MT TrackBack page for OSXCon to see the few folks who are blogging. There's what, four of us? Maybe five? :-(
I realized something the other day. I've been programming in Perl for roughly seven or eight years now. Maybe nine. It's hard to remember for sure. But at some point it lost its shine. It used to be a lot of fun. I programmed for the hell of it. I loved the Perl culture and community. I didn't dream of using anything else.
Anymore, I'm really seeing Perl as just another language/tool/whatever. I feel compelled to start looking into other languages--namely Ruby and Python, just to stimulate that part of my brain and have some fun again.
I don't know when this all happened, but it has really only come to my attention recently. I suppose it's natural. I mean, how long can you program in the same language before it get a little old?
Or maybe it's part of something larger that I haven't figured out yet.
The business news of the day has long been telling us that we're in a very slow economic period. Yet I've received multiple job solicitations recently. Not random head-hunters or anything. People I know, who know me relatively well, want me to work for/with/near them.
I'm not sure what to make of it all. But one thing I've learned is that sometimes life is trying to tell you something. And it's best to try and listen. Yet other times, you're reading something into nothing. Hard to tell which is the case here, but I'm leading toward the former.
The really odd part is that I hadn't even touched my resume until the other night when someone asked me to send a copy. So I updated it a bit. No harm done. I don't know what, if anything, will come of all this.
Josh, a former Yahoo, points to Business 2.0's The CEO From Another Planet story and says "Let's just say that this article couldn't have been more accurate."
Well, I'm not going to defend Terry. He doesn't need me to do that. It's pretty clear, things are very different at Yahoo than they were a few years ago. Or even one year ago. Semel has a lot to do with that. He sets (or should set) the tone for the company. A lot of that article is true, as far as I can tell. But I'm just a lowly engineer. I have no idea what's going on at the top or near the top. Is that, itself, a problem? Perhaps.
On the plus side, I can say that there are several things coming "soon" that will hopefully demonstrate how Terry's ties to Hollywood can finally benefit Yahoo in a big way. But only time will tell.
In the meantime, Yahoo continues the transformation from a company that changed the world to a company that's just like every other big (public) company.
*sigh*
On a related note, I'm still amazed that Filo and Jerry are still involved in the company. Those guys are amazing and in may ways, they're still the heart and soul of Yahoo. How many other Internet companies can say that?
This talk is not supposed to be technical or marketing, but more of a ramble. We'll see. :-)
"Living above the Curve" BB is a strange company, as they've been Mac-only are have been in business for over 10 years. Haven't gone out of bussiness, gone public, or been bought. It's very rare. They make a living doing Mac-only. So it can be done.
What are the topics: Why are they still around? Embracing the hard. World domination.
Why are they still around? Customer loyalty. They spend the money to make it all possible. And those customers help to breed other customers. Word of mouth, new features, etc. Industry changes have helped, too. BBEdit was originally a programmer's editor. But BBEdit had a plug-in interface. And HTML came along. Customers began developing HTML plug-ins. People actually bought BBEdit for the purpose of using one of those 3rd party plug-ins.
Back in 1997, things were bad. Everyone thought Apple was gonna die. Other companies began developing Windows software. And that's often more expensive. And if your Windows software is really good, Microsoft eventually buys you, puts you out of business, or otherwise squashes you.
Mac platform changes have helped too. 68k to PowerPC helped a lot. Allowed BB to adopt very quickly. The move to OS X has helped too. If you move quickly (like BB did), you can ride the wave very well.
They're careful about picking their fights very carefully.
There is always room for 3rd party software. Apple can't do it all and needs 3rd party software. They always will.
What about the move to OS X and "carbonizing" their application? OS X is brutal because it doesn't tolerate mis-behaving applications. OS 9 let things gradually degrade. OS X is WAY more stable. Cabon forced them to review their code and find latent bugs that had been around for years and years. Had to work to meet Aqua standards. Had to realize that OS X is Unix and that means a whole lot of new things they could do.
Don't limit yourself to Carbon. Use Cocoa. Use Unix functionality.
BBEdit can be run from the command-line and take command-line args. You get things file filename globbing automatically. You can pipe to it. It even has a man page! Very cool. :-)
There's a feature that acts a lot like a shell buffer in Emacs. You even get undo. You can run arbitrary Unix filters on selected text.
Java has been successful everywhere but the desktop. Or at least that's what people here. It's big on the server and that downs out the desktop news.
What else went wrong?
Applets.
Microsoft's battle with Java.
For the longest time, there really has been only ONE desktop. It was the only market that mattered for developoers.
We're finally getting desktop competition from Linux and Mac OS X.
A lot of the cool things about OS X are the little things. Like closing the notebook lid.
Why use Java?
Productivity. Lots of APIs. Tools. Community. Security and reliability. End-to-end story. Portable skills. Lots more. Memory management and platform neutrality.
Swing API. It's the 747 cockpit of UI toolkits. Lots of components on jars.com, etc. Very extensible.
Swing problems. Complexity. It's easy, but finding the easy way is hard. You need to watchout for performance pitfalls too.
Good stuff. Abstract data models. Map data structures to models.
And then you've got Java2D (like Postscript/Quartz/etc), Java3D, Advanced imaging, Media, Networking, and so on.
Used NetBeans IDE to show us the source to his Java-based presentation system. (Very cool, but I note a distinct lack of comments in his code.) It does reformatting on the fly--rather quickly. Makes good use of threading.
Lots of screen shots and demos. (But very little OS X specific stuff. I'm sorta disappointed.)
Which was the better UI? Mac OS 9? Mac OS X? Windows? Nobody can agree. Tim O'Reilly didn't like Mac OS 9. He gets OS X. Traditional Mac users are a bit annoyed by OS X. They think it's a bit too much like Windows.
What about Intel hardware? Jordan can't say (of course). A lot of folks are willing to pay for the extra Mac hardware. But in the larger world, people worry about pricing, Morotola, and so on.
Existing Mac users need Quark on OS X. It's supposed to be coming someday.
An audience member asks about performance improvement. Jordan says they're still working quite hard on it. They need to catch up with Linux. Mac user's don't buy new hardware every year. They keep their boxes *forever*. So the software really does have get faster.
Jordan: "One of the benefits of going with Morotorla is that we don't assume the hardware will get faster."
Lots of laughter. :-)
Discussion about what might be tuned. How to keep the foreground processes faster. Scheduling is very tricky.
What about having X installed by default?
Jordan thinks it *could* happen someday. Or at least bundle the client libraries, etc. Or maybe put it on the developers CD if not installed by default.
Notice how Apple has made sure that normal Mac users NEVER have to open the terminal if they don't want to.
DarwinPorts stuff coming soon. Stay tuned for news. Watch DaemonNews for more info.
There are Open Source developers still don't trust Apple yet. What if their whims change? Might developers be stranded--even a concern for in-house corporate development. What can be done?
Tim says that corporations often do orphan products. That's life. But in the Open Source world, folks can pick it up and keep it alive. Might Apple give products away when they orphan them? Jordan will take it back to Apple.
OS X is a safe bet. Apple is all over it. They are being very consistent, focused, and determined. They're in it for the long-haul.
Lots of talk about iApps. Will they be more open? Plugins and scriptability? The development teams seem very responsive.
Nat brings up the "culture shock" involved in coming from the Unix world. Things like, "wow, I have to pay for all my software." What can Apple do to help ease the transition. The other is that you can' fix apps, but you can usually talk to someone who can. That can be frustrating.
Jordan stressing sending in feedback. Don't be so cynical. Apple is not Sun or IBM.
MacDevCenter.com agrees.
Complainers need to register as developer and FILE BUG REPORTS, just like in the Open Source world.
Linux switchers want cheaper hardware.
Someone asks Jordan what it was like as an Open Source guy at Apple. He got there late enough that he didn't have to fight a lot of battles. He got lucky.
Ack! TiBook power is low. Must submit and go off-line soon. Should have charged it last night.
Talking about his mainframe background and roots in computing. He played a lot on the big iron and sorta "missed" the PC revolution. He wasn't terribly interested in PCs for quite a while. Then he got to play with one of the very first Apple II computers. But he liked the mainframe hardware yet.
In 1980, he began working with Unix and started to understand and appreciate the design of Unix.
In the old days, your OS came from your hardware vendor. It was all about lock-in. If you let them interoperate, they might switch to another vendor. Hide the internals. Make it a bit of a pain to deal with--but not too much. The OS was specific, often designed for a particular market and task.
The hardware was very big and expensive (of course). There was no portability. They were always multi-user and the displays were dumb terminals.
Lots of talk about Unix history, AT&T, and Berkely.
Berkeley added long filenames, virtual memory, curses, vi, job control, UFS filesystem, and so on. SunOS and DEC Untrix came along. The wars started when AT&T and Sun got together. The others started the Open Software Foundation (OSF). OSF1 vs. SysV. Yuck. SysV vs. BSD too.
BSD ended up under a "legal cloud" for quite a while.
Nobody wrote portable software. Everyone was pissed at each other. They all created their own windowing/desktop systems. Intel didn't rule the CPU space yet, so the world was very fragmented on the architecture front. That's why there's so much portability and ./configure stuff floating around today.
Then came the GUI wars. Sun had NeWs, which was based on Postscript. But it lost to X. So they created OpenLook, which battled with OSF's Motif. That made Unix softwrae more expensive, and Unix lost the desktop war long ago as a result. Windows was unified but Unix was not. CDE vs. OpenView vs. ... Ugh.
Every vendor tried to differentiate themselves with desktops. So most software vendors just went to Windows.
X is the reason that Unix lost the desktop war. It was a system by engineers, for engineers. Text and font handling sucked (and still does). No printing support. No multi-media support. The APIs are all very low-level. Everyone had to write their own printing engine.
Now you know why we have Qt vs. Gtk vs. whatever. It should have been decided long ago.
Unix users are often averse to solving "big picture" problems. Unix was assumed to be user-unfriendly. The only real API, libc, was very low-level. Folks had to write their own APIs. Again, look at Qt vs. Gtk. Everyone wanted to own the standard, so nobody did.
But Unix remained alive. Why?
The underlying tool-building philosophy is a Good Idea. Open systems are very compelling. There are some very smart people in the Unix community. Unix was open to inspection and extension.
Unix has risen again. Why?
The Internet. Unix TCP/IP kicks ass. It all started on Unix and still happens first on Unix. Unix handles load, so it works on servers well. And there was always an installed based in the science and tech communities. Open Source broke the vendor lock. And Linux Torvalds as a poster child has done a great job. He beat the odds. The BSD lawsuit finally went away. BSD was freed.
Unix and BSD both made a comback. It's now a good word in the IT circles again. It became a foundation for the next layer: Perl, Apache, Python, Emacs, GCC, other GNU stuff.
But what about the desktop?
It had been lost to Microsoft. Until Apple came along and tried to fix that with OS X. Mach microkernel, threading, power management (for portables), ease of use, etc. To get the developers back, they've gotta provide great APIs. That's what Cocoa is all about. And the Java implementation works very, very well. They haven't sacrificed multimedia support either. Their internationalization support it world-class.
Quartz is cool. Finally solves the font/bitmap/display/printing problems on Unix. Makes good rendering and printing work well. It's all built-in. OpenGL for industry standard 3D and it works out of the box correctly. That's important for the game industry. Quartz Extreme unified 2D and 3D and makes good use of the GPU. (I guess that's important when your CPUs are still slower than Intel's?)
Great video support, including QuickTime 6.
Putting all that together, now Unix desktops are getting the media apps they only dreamed about before. Apple has done it *and* commoditized it too. It's not like paying SGI prices anymore.
And, if you still want X, you can get X. It's there for those who want it. The XFree86 project has done a great job.
So...
Mac OS X can help try to win (some of) the desktop back. It is getting the critical mass necessary to get ISVs back building software for it. And the cool hardware and tools is bringing back many developers too.
And, well, it's NOT Windows. :-)
Now for the panel discussion.
Did anyone else catch the Warchalking story on Marketplace today?
I'm a bit puzzled that the approached it as a "hacking" story. They don't get it, either, I guess.
*sigh*
Wow. Check out Western Digital's Drivezilla. They've announced 200GB hard disks. Something tells me that I'll have my own terabyte by the end of 2003.
I had to leave the conference after the first half of the day so that I can get some stuff done at work and some work on the book.
David Pogue's keynote was excellent. He knows the Mac, Apple history, and is very funny. I highly recommend seeing him speak if you get the chance.
Tim O'Reilly's keynote was disappointing for me. If you've been paying attention to what Tim's been saying in the last year, you didn't miss anything. A lot of it has come up in his weblog and in his other presentations. He also ended up going over the alloted time and had to really cut himself short. I never really seen that happen to him before.
I attended a presentation called Automating Mac OS X by Matt Neuburg of TidBITS. It was interesting at a high level. I never realized how AppleScript and Apple Events work and what they're really capable of. He showed a bit of Real Basic and Cocoa as well as some Perl, Python, and Ruby--all on OS X, of course. It's interesting to see how many applications are "externally scriptable" on OS X and how easy it is.
He even had a demo that involved using Manila to build an on-line photo gallery. Neat stuff.
That's it for now. Sorry I can't cover more of today.
I'll be headed back tomorrow to listen to Jordan Hubbard's morning keynote about OS X and Unix. I'll also cover James Gosling's afternoon keynote on OS X and Java. An Apple exec was very happy to tell me (several months ago) that Gosling now uses OS X on his desktop and that Java is great.
I haven't completely decide which other sessions I'm going to hit, but I'm hoping to nail at least two of them. We'll see.
David Pogue is talking about Mac in the past, present, and future.
Mac market is growing. It's a small part of a very large and growing pie. So Apple isn't dead and won't die. But they'll never be the big player.
The history comes first. The two Steves. They had color first. And it was the first beige machine! Heh. Steve Jobs visits PARC and gets some ideas from them. Saw the mouse, menus, and more. Decided they belonged on his computer. (Heh, Xerox invented the brain-dead business plan. They never did anything with the ideas.) The Lisa was born.
Then comes Scully, fresh from selling sugar water. It was the reality distortion field at work. Jobs went to work on the Mac project. And MacWorld magazine was born not long after. It was all GUI all the way.
Cool. System 1 screenshots. 128k and floppy. Heh.
Scully sends Jobs away and strips his responsibility. Goes off to start Next. Scully does layoffs and Windows 1.0 comes out to compete with Apple. The first color Mac comes out, as does the Newton. It flops big time.
4,000 newton modems make for great dominos.
Then came Spindler and Amelio. More Macs came out. They were all pretty boring.
Apple bought Next and got Steve back. Then came the new Macs. Flavors, cubes, and more. Lots of OS revs. Then came OS X.
And it was good.
Talks lots about all the new features in OS X.
What about the future? What'd be nice?
Icon labels, location manager, file encyrption, ram disk, Windows "send to" menu, virtual desktop (yes! I'd love that), and so on. Networking assistant would be good too. Voice and Video conferencing.
What Windows could learn from the Mac. Taste. AppleScript. Freedom from "activation". Driver invisibility. Text-to-speach. Centralized control panel. Drag-n-drop install. And uninstall. Coherent app names. One menu for software apps. The list goes on and on.
What about OS XV and beyond?
Here are some ideas. Document aging. Icons that reflect their contents. Big docs get "taller stack" icons. Screen memorizer. Broadband software. eBay for documents (clip art, music, info, etc).
RAM growth. Disk growth. Much more powerful CPUs, of course. 55 inch monitors! :-)
Seriously, CRTs are dead. Very dead. Bluetooth is a sure thing. It's going to make a big difference. Speech recognition is not the answer. Keyboards are here to stay. Battery problems won't go away for a long time.
RAM drives are coming. The USB kind. Very cool.
DVDs. CDs will go. Microsoft is here to stay too.
Windows wizards could just keep getting worse and worse.
Apple will continute to set design trends, like the iPod and the new iMac.
Tim O'Reilly is giving his keynote right now. Talking a lot about watching
Alpha Geeks, wireless netowrk, bots, and so on. Here are some notable items
that Tim mentions.
"The future is here, it's just not evenly distributed yet" -- William
Gibson.
Hackers push the envelope, and entreprenurs make things eaiser for
ordinary users.
The evolution from CGI to PHP/mod_perl and on to GUI site builders.
Eventually, consolidation happens around technologies as mature. We
need to balance control and innovation. Microsoft is probably too closed
and controlling. It's an ecosystem. Don't pollute it.
Paradigm shifts and distruptive technologies.
What's to like about OS X:
Piracy is progressive taxation.
Thanks to an anonymous helper, I've mirrored all 5 ISO images for Red Hat 8.0.
Ohio - http://family.zawodny.com/~jzawodn/iso/rh80/ (DS-3 connection via Sprint)
California - http://litterbox.zawodny.com/~jzawodn/iso/rh80/ (multiple gigabit connections)
Now be nice. :-)
Watch out Yahoo Shopping and other shopping meta-sites. Google is at it again with Google Catalogs. The only question in mind mind is this: Will it hit big this Christmas shopping season or next year?
Google Product Search and Ratings. Imagine the possibilities.
An article titled Mac poses as much of challenge to Linux as to Windows is both insightful and missing the point.
The general thrust of the article, is dead-on. It's just the sort of stuff I've been saying to folks about OS X. It's the only "Unix on the Desktop" package that really delivers. (Side note: I'm going to be at O'Reilly's OS X Conference tomorrow.) Red Hat is playing catch up with Apple and they're at a severe disadvantage.
However, later in the article is a mini-review of Red Hat Linux 8.0, which just came out today. The author notes that:
In short, a PC loaded with Red Hat 8 can handle about 95 percent of the average user's needs. This much software from Microsoft would cost about $400. But Red Hat 8 can be downloaded from the Internet for free. To get service and support, you have to pay for the software, but only $39.95 for the basic desktop version.
Excellent. Many folks have been predicting that this day would come.
The article also says:
In our test of Red Hat 8, none of this mattered. The disk partition process is automated now. As for compatibility, the software identified and self-installed all the vital peripherals on our test machine without incident. And there was none of this Microsoftian nonsense about software authentication. As a result, Red Hat is actually easier to install than Windows XP.
That's no surprise. Red Hat has been working themselves crazy to ensure that it'd be easy to install. Easier than Windows. Easier than any other version of Linux. Easier than any commercial Unix.
But then we get to this:
However, cheap is not enough. Red Hat 8 is still a ragbag of code, written by amateurs and part-timers and jumbled all together. And it shows. For instance, the feature for allowing you to share information with a Handspring handheld computer didn't work properly. A Red Hat engineer admitted that he wasn't sure of the reason, because the underlying software was written by somebody outside the company and he knew little about it.
Corporate users pay for Microsoft products largely because they get a decent level of support from the company. Because Linux isn't controlled by Red Hat or any other company, delivering the same level of service is far harder. But it'll have to be done.
This shows that folks at the Boston Globe, like so many other of the mass media STILL DON'T GET IT.
Linux is an operating system. It is not a desktop. Red Hat Linux is a large bundle of software, little of which was written by Red Hat. To compare all the pieces of Red Hat Linux that Red Hat did not develop with Microsoft Windows simply doesn't work. It's apples and oranges. Stop it. All of you.
Linux and Open Source are a new way of doing things. They are not the commercial software market. The whole point is that this "ragbag of code, written by amateurs and part-timers and jumbled all together" is often (certainly not always) better than the commercial alternatives.
Just look at what Apache has done in the server world. It's the number one web server and has been for a long, long time. Microsoft is not winning in that space. Expect to see similar things on the desktop. I'm not predicting that Linux will take over the desktop--not by a long shot. But there will be killer apps on the desktop. Just look at all the innovative new projects that have sprung up to build on the foundation that Mozilla has provided.
[UPDATE: There's important new information in this article.]
This is an issue that comes up all the time at work. It is an issue for roughly four reasons:
This is an attempt to answer that question for folks at work and elsewhere.
I'll start with the simple answer.
If you don't have a preference, use Linux. You'll be happier. Trust me.
If, on the other hand, you'd like to stick with FreeBSD for some reason (and there are good reasons--one is noted later), read on.
Having said that, let's look at the issues a little more closely. FreeBSD is a great operating system, but it has one important weakness that MySQL is very good at highlighting--threading support. FreeBSD's threads implementation isn't very good. I won't say that it sucks, because it could be a lot worse.
Threads on Linux are created using the clone() call, which is similar to fork(). You end up with a separate process in the process table, but the memory is shared among the processes. This means that the kernel gets involved in scheduling. From the kernel's point of view, they're all just processes. Many folks refer to this as kernel threading even though it's different than what the Solaris kernel does (for example). Some call that real kernel threading. :-)
Anyway, the LinuxThreads FAQ goes into a bit more detail. LinuxThreads is a library, available for other platforms--including FreeBSD. We'll come back to that in a bit.
FreeBSD implements user-level threads. That means the kernel isn't aware of the threads and doesn't get involved in scheduling. Instead all the work is done in user space rather than kernel space. When your run top or ps on a machine that does this, unlike like in Linux, you'll see a single process rather than one per thread.
This is discussed a bit here in relation to LinuxThreads, which we'll get to.
Note that in FreeBSD 5.x, this may all be fixed. Time will tell. There is lot of working going on in the area of threading and kernel scheduling for FreeBSD 5.x.
Having run MySQL on various flavors of FreeBSD for the last 2.5 years, I can say that it has been a bumpy ride at times. Versions older than 4.2 (or maybe 4.3) have serious problems. I had a test case that could kill a MySQL server running on older versions of FreeBSD in a matter of minutes.
On more modern FreeBSD, things are better but not perfect. All the problems we've encountered at Yahoo seem to fall into 4 buckets.
1. Non-thread safe DNS Lookups
Certain operations are not thread-safe on FreeBSD. A fine example of that is gethostbyname(), which MySQL calls to convert host names in to IP addresses. Usually this happens for each new connection to the server and whenever MySQL needs to contact another machine--typically a replication slave connecting to its master.
Based on our testing, the only truly safe way to operate is to use the --skip-name-resolve flag for starting mysqld AND specifying the IP address of the master instead of the hostname. That virtually eliminates the need for MySQL to call gethostbyname().
The symptom of this problem is that the mysqld will consume all the available CPU time even when there are few (if any) queries running. You can try and kill -6 the mysqld process and then run it thru gdb to get a backtrace. You'll likely see something like this:
#0 0x829c94c in _thread_kern_sched_state_unlock () at ./cp/tinfo2.cc:300 #1 0x829c0e0 in _thread_kern_sched () at ./cp/tinfo2.cc:300 #2 0x829c787 in _thread_kern_sched_state () at ./cp/tinfo2.cc:300 #3 0x82c5fdc in kevent () at ./cp/tinfo2.cc:300 #4 0x82c5a4f in res_send () at ./cp/tinfo2.cc:300 #5 0x82a4308 in res_query () at ./cp/tinfo2.cc:300 #6 0x82a4737 in res_querydomain () at ./cp/tinfo2.cc:300 #7 0x82a44bb in res_search () at ./cp/tinfo2.cc:300 #8 0x82a9a00 in _gethostbydnsname () at ./cp/tinfo2.cc:300 #9 0x82a8739 in gethostbyname2 () at ./cp/tinfo2.cc:300 #10 0x82a86d4 in gethostbyname () at ./cp/tinfo2.cc:300 #11 0x8275fc4 in my_gethostbyname_r ( name=0x1b5f79a8 "your_hostanme", result=0x9fa659b8, buffer=0x9fa651b8 "\032", buflen=2048, h_errnop=0x9fa651b0) at my_gethostbyname.c:108 #12 0x80d6fbd in mc_mysql_connect () #13 0x80d6b37 in mc_mysql_reconnect () #14 0x80d4506 in safe_reconnect () #15 0x80d3fb8 in handle_slave () #16 0x828ffa5 in _thread_start () at ./cp/tinfo2.cc:300 #17 0x0 in ?? ()If you see that, get rid of DNS lookups.
2. Unfair Scheduling
We've seen instances when a single MySQL thread doing a lot of I/O work (deleting a lot of rows from a table) seems to monopolize all the CPU time. When this happens, even the most trivial SELECT queries against unrelated tables can take a long time or even block until the heavy I/O work is complete. It feels like somehow the I/O thread is unfairly getting more of the CPU time.
3. High Load
Even without the other two problems, I've seen MySQL servers on FreeBSD start to act strangely under real stress--meaning a reasonable number of clients (30 or more) that are really pounding on MySQL. If I had a test case that could always reproduce it, I'd certainly make it available. But the fact is that it seems fairly random.
Luckily most of our MySQL installations don't have that problem because they're not hit really hard or they're not hit that hard for very long. If I had to guess, I'd say that 95% of folks never see this problem. The remaining %5, of course, are rather upset when they do.
4. No SMP Support
Because the threads are not managed by the kernel, there's no way to make use of multiple CPUs. The scheduler can't get two threads of the same process running on two CPUs at the same time. So you're limited to either running on single CPU machines or running multiple instances of MySQL on the same physical machine--and that involves some interesting management problems that I'd rather not deal with.
5. Missing Locks
This may somehow be related to problem #2. We've seen cases where several threads want to run queries against a single MyISAM table. When one thread is doing a lot of work, such as a massive delete, all the readers wait for a shared lock on the table. In order for that to happen, the writer needs to finish and release its exclusive lock. The strange thing is that sometimes the writer finishes and even disconnects, but the readers are all stuck waiting for a lock according to SHOW FULL PROCESSLIST.
When that happens, nobody can query the table. It's effectively off-limits. It's as if all the readers somehow "missed" the fact that the writer is done. Or maybe the writer's lock didn't get released properly. The only solution is to kill all the locked threads. Once that happens, it will usually begin to work normally.
The most popular solution is to recompile MySQL and link it with the LinuxThreads library. Doing so gives you the benefits of kernel assisted "threading" (fair scheduling, SMP, and not needing thread-safe gethostbyname()).
Sounds perfect, right? Not exactly. It turns out that this does solve those problems quite well. But I've found at least one thing that stopped working with a LinuxThreads version of MySQL on FreeBSD and another occasional but very annoying problem.
MySQL's wait_timeout setting can be used to automatically close the connection (and terminate the thread) of a client who as been idle long than N seconds. It simply stops working with LinuxThreads. I don't know why (yet).
The more important bug is rather mysterious. Every once in a while, a LinuxThreads-enabled MySQL 4.0.x server will decided that no databases exist anymore. All clients will be rejected with an "Unknown database 'foo'" message (where "foo" is the name of the database you'd like to connect to ). The SQL slave thread stops. The only solution appears to be restarting MySQL. It has memory available. I've tried adjusting the table cache and other variables that might seem related, but it hasn't helped. No helpful messages are logged in the error log.
Using a snapshot of the exact same code, compiled on Linux, I never see that problem. Ever.
My guess is that the problem is related to load, but I cannot reliably reproduce it (yet?).
Other than those two problems, it seems to work quite well. But I've only been running it for a few months with alpha and beta versions of MySQL. I'm fairly sure that all the other odd problems I've had with them are not related to LinuxThreads, but I can't yet say so with 100% certainty.
From what I've seen so far, using LinuxThreads ought to help even on single-CPU FreeBSD boxes.
If you are in the majority of FreeBSD users who never see major problems with MySQL on FreeBSD, there are some real advantages to staying there--not moving to Linux. The single biggest advantage is FreeBSD's VM subsystem. Under the workloads I've tested, it always outperforms Linux 2.4.x (where x is 9, 12, 16, and 18). When Linux decides to swap out data that is destined to be used again soon (think key buffer), FreeBSD doesn't go down that route. See my earlier blog entry for more details.
Linux doesn't have the problems that FreeBSD does. Threading works quite well. On the other hand, its VM can be a bit dumb at times. Turning off swap makes that non-issue if it does become a real problem.
Linux is still my preferred choice.
If anyone is able to assist in further debugging the FreeBSD problems--especially the LinuxThreads problems, I'd love the help.
I like Kasia's latest idea. Let's all try that. But not on the same day. Maybe we should all use our birthdays as the basis. Since I was born on June 4th, I'll be stupid on the 4th of every month.
Heh. I fear that it would get old after a few months. Maybe. But it'd be fun while it lasted, and it might even make a point.
I forgot to write this up yesterday and didn't feel like forging the blog entry date. Anyway, while taking a break from writing the book, I tried to finish up the work on friends.zawodny.com before I ship it off to get racked at WCNet where family.zawodny.com (which hosts jeremy.zawodny.com) has sat for about two years.
The new machine has 2 80GB Maxtor disks, one on each IDE channel. I have four RAID-1 filesystems (using Linux software RAID) on the drives: /, /usr, /var, and /home. I did some tinkering around and, after a few Goggle searches, I managed to get it to boot from either disk in the event of a failure. When hda failed (I unplugged it), the system booted from hdc just fine. And when hdc failed, it booted just fine from hda and ran in "degraded" mode until I rebooted with both disks and used raidhotadd to re-add the previously failed partitions to each RAID volume.
This is excellent. I had trouble getting this to work on machines in the past and opted for simply having the data on RAID volumes but no fail-over for booting. Now that I've seen it work, I see no need to ever really bother with hardware RAID. The built-in stuff works great.
I should note that I had tried to out-smart things (using the information like that found here) but it never worked for me. Once I tried to just follow the directions and told LILO that I wanted to use /dev/md0 rather than /dev/hda, it Just Worked. Excellent!
Combine all that with the fact that the Debian Woody install CD allows me to select from ext2, ext3, and ReiserFS at installation time, it's now far easier to build a robust Linux server that it used to be.
Strangely, I managed to get up before 9am (on a Sunday!) without the aid of an alarm clock. Not sure how that happened. I had some breakfast (muffin with jelly and peanut butter, and strawberry drink) while catching up on e-mail (not much) and blogs (a little more).
I had NPR on for background noise and because I often listen to Weekend Edition on KQED FM.
While listening, I got to hear a great discussion with Peter Gabriel about his new album: Up. I really enjoyed the piece and was thrilled to hear that his first studio album in 10 years is now available. I've been a big fan for a long, long time. I've already ordered it and hope share my thoughts on it later this week.