Watch out Yahoo Shopping and other shopping meta-sites. Google is at it again with Google Catalogs. The only question in mind mind is this: Will it hit big this Christmas shopping season or next year?
Google Product Search and Ratings. Imagine the possibilities.
An article titled Mac poses as much of challenge to Linux as to Windows is both insightful and missing the point.
The general thrust of the article, is dead-on. It's just the sort of stuff I've been saying to folks about OS X. It's the only "Unix on the Desktop" package that really delivers. (Side note: I'm going to be at O'Reilly's OS X Conference tomorrow.) Red Hat is playing catch up with Apple and they're at a severe disadvantage.
However, later in the article is a mini-review of Red Hat Linux 8.0, which just came out today. The author notes that:
In short, a PC loaded with Red Hat 8 can handle about 95 percent of the average user's needs. This much software from Microsoft would cost about $400. But Red Hat 8 can be downloaded from the Internet for free. To get service and support, you have to pay for the software, but only $39.95 for the basic desktop version.
Excellent. Many folks have been predicting that this day would come.
The article also says:
In our test of Red Hat 8, none of this mattered. The disk partition process is automated now. As for compatibility, the software identified and self-installed all the vital peripherals on our test machine without incident. And there was none of this Microsoftian nonsense about software authentication. As a result, Red Hat is actually easier to install than Windows XP.
That's no surprise. Red Hat has been working themselves crazy to ensure that it'd be easy to install. Easier than Windows. Easier than any other version of Linux. Easier than any commercial Unix.
But then we get to this:
However, cheap is not enough. Red Hat 8 is still a ragbag of code, written by amateurs and part-timers and jumbled all together. And it shows. For instance, the feature for allowing you to share information with a Handspring handheld computer didn't work properly. A Red Hat engineer admitted that he wasn't sure of the reason, because the underlying software was written by somebody outside the company and he knew little about it.
Corporate users pay for Microsoft products largely because they get a decent level of support from the company. Because Linux isn't controlled by Red Hat or any other company, delivering the same level of service is far harder. But it'll have to be done.
This shows that folks at the Boston Globe, like so many other of the mass media STILL DON'T GET IT.
Linux is an operating system. It is not a desktop. Red Hat Linux is a large bundle of software, little of which was written by Red Hat. To compare all the pieces of Red Hat Linux that Red Hat did not develop with Microsoft Windows simply doesn't work. It's apples and oranges. Stop it. All of you.
Linux and Open Source are a new way of doing things. They are not the commercial software market. The whole point is that this "ragbag of code, written by amateurs and part-timers and jumbled all together" is often (certainly not always) better than the commercial alternatives.
Just look at what Apache has done in the server world. It's the number one web server and has been for a long, long time. Microsoft is not winning in that space. Expect to see similar things on the desktop. I'm not predicting that Linux will take over the desktop--not by a long shot. But there will be killer apps on the desktop. Just look at all the innovative new projects that have sprung up to build on the foundation that Mozilla has provided.
[UPDATE: There's important new information in this article.]
This is an issue that comes up all the time at work. It is an issue for roughly four reasons:
This is an attempt to answer that question for folks at work and elsewhere.
I'll start with the simple answer.
If you don't have a preference, use Linux. You'll be happier. Trust me.
If, on the other hand, you'd like to stick with FreeBSD for some reason (and there are good reasons--one is noted later), read on.
Having said that, let's look at the issues a little more closely. FreeBSD is a great operating system, but it has one important weakness that MySQL is very good at highlighting--threading support. FreeBSD's threads implementation isn't very good. I won't say that it sucks, because it could be a lot worse.
Threads on Linux are created using the clone() call, which is similar to fork(). You end up with a separate process in the process table, but the memory is shared among the processes. This means that the kernel gets involved in scheduling. From the kernel's point of view, they're all just processes. Many folks refer to this as kernel threading even though it's different than what the Solaris kernel does (for example). Some call that real kernel threading. :-)
Anyway, the LinuxThreads FAQ goes into a bit more detail. LinuxThreads is a library, available for other platforms--including FreeBSD. We'll come back to that in a bit.
FreeBSD implements user-level threads. That means the kernel isn't aware of the threads and doesn't get involved in scheduling. Instead all the work is done in user space rather than kernel space. When your run top or ps on a machine that does this, unlike like in Linux, you'll see a single process rather than one per thread.
This is discussed a bit here in relation to LinuxThreads, which we'll get to.
Note that in FreeBSD 5.x, this may all be fixed. Time will tell. There is lot of working going on in the area of threading and kernel scheduling for FreeBSD 5.x.
Having run MySQL on various flavors of FreeBSD for the last 2.5 years, I can say that it has been a bumpy ride at times. Versions older than 4.2 (or maybe 4.3) have serious problems. I had a test case that could kill a MySQL server running on older versions of FreeBSD in a matter of minutes.
On more modern FreeBSD, things are better but not perfect. All the problems we've encountered at Yahoo seem to fall into 4 buckets.
1. Non-thread safe DNS Lookups
Certain operations are not thread-safe on FreeBSD. A fine example of that is gethostbyname(), which MySQL calls to convert host names in to IP addresses. Usually this happens for each new connection to the server and whenever MySQL needs to contact another machine--typically a replication slave connecting to its master.
Based on our testing, the only truly safe way to operate is to use the --skip-name-resolve flag for starting mysqld AND specifying the IP address of the master instead of the hostname. That virtually eliminates the need for MySQL to call gethostbyname().
The symptom of this problem is that the mysqld will consume all the available CPU time even when there are few (if any) queries running. You can try and kill -6 the mysqld process and then run it thru gdb to get a backtrace. You'll likely see something like this:
#0 0x829c94c in _thread_kern_sched_state_unlock () at ./cp/tinfo2.cc:300 #1 0x829c0e0 in _thread_kern_sched () at ./cp/tinfo2.cc:300 #2 0x829c787 in _thread_kern_sched_state () at ./cp/tinfo2.cc:300 #3 0x82c5fdc in kevent () at ./cp/tinfo2.cc:300 #4 0x82c5a4f in res_send () at ./cp/tinfo2.cc:300 #5 0x82a4308 in res_query () at ./cp/tinfo2.cc:300 #6 0x82a4737 in res_querydomain () at ./cp/tinfo2.cc:300 #7 0x82a44bb in res_search () at ./cp/tinfo2.cc:300 #8 0x82a9a00 in _gethostbydnsname () at ./cp/tinfo2.cc:300 #9 0x82a8739 in gethostbyname2 () at ./cp/tinfo2.cc:300 #10 0x82a86d4 in gethostbyname () at ./cp/tinfo2.cc:300 #11 0x8275fc4 in my_gethostbyname_r ( name=0x1b5f79a8 "your_hostanme", result=0x9fa659b8, buffer=0x9fa651b8 "\032", buflen=2048, h_errnop=0x9fa651b0) at my_gethostbyname.c:108 #12 0x80d6fbd in mc_mysql_connect () #13 0x80d6b37 in mc_mysql_reconnect () #14 0x80d4506 in safe_reconnect () #15 0x80d3fb8 in handle_slave () #16 0x828ffa5 in _thread_start () at ./cp/tinfo2.cc:300 #17 0x0 in ?? ()If you see that, get rid of DNS lookups.
2. Unfair Scheduling
We've seen instances when a single MySQL thread doing a lot of I/O work (deleting a lot of rows from a table) seems to monopolize all the CPU time. When this happens, even the most trivial SELECT queries against unrelated tables can take a long time or even block until the heavy I/O work is complete. It feels like somehow the I/O thread is unfairly getting more of the CPU time.
3. High Load
Even without the other two problems, I've seen MySQL servers on FreeBSD start to act strangely under real stress--meaning a reasonable number of clients (30 or more) that are really pounding on MySQL. If I had a test case that could always reproduce it, I'd certainly make it available. But the fact is that it seems fairly random.
Luckily most of our MySQL installations don't have that problem because they're not hit really hard or they're not hit that hard for very long. If I had to guess, I'd say that 95% of folks never see this problem. The remaining %5, of course, are rather upset when they do.
4. No SMP Support
Because the threads are not managed by the kernel, there's no way to make use of multiple CPUs. The scheduler can't get two threads of the same process running on two CPUs at the same time. So you're limited to either running on single CPU machines or running multiple instances of MySQL on the same physical machine--and that involves some interesting management problems that I'd rather not deal with.
5. Missing Locks
This may somehow be related to problem #2. We've seen cases where several threads want to run queries against a single MyISAM table. When one thread is doing a lot of work, such as a massive delete, all the readers wait for a shared lock on the table. In order for that to happen, the writer needs to finish and release its exclusive lock. The strange thing is that sometimes the writer finishes and even disconnects, but the readers are all stuck waiting for a lock according to SHOW FULL PROCESSLIST.
When that happens, nobody can query the table. It's effectively off-limits. It's as if all the readers somehow "missed" the fact that the writer is done. Or maybe the writer's lock didn't get released properly. The only solution is to kill all the locked threads. Once that happens, it will usually begin to work normally.
The most popular solution is to recompile MySQL and link it with the LinuxThreads library. Doing so gives you the benefits of kernel assisted "threading" (fair scheduling, SMP, and not needing thread-safe gethostbyname()).
Sounds perfect, right? Not exactly. It turns out that this does solve those problems quite well. But I've found at least one thing that stopped working with a LinuxThreads version of MySQL on FreeBSD and another occasional but very annoying problem.
MySQL's wait_timeout setting can be used to automatically close the connection (and terminate the thread) of a client who as been idle long than N seconds. It simply stops working with LinuxThreads. I don't know why (yet).
The more important bug is rather mysterious. Every once in a while, a LinuxThreads-enabled MySQL 4.0.x server will decided that no databases exist anymore. All clients will be rejected with an "Unknown database 'foo'" message (where "foo" is the name of the database you'd like to connect to ). The SQL slave thread stops. The only solution appears to be restarting MySQL. It has memory available. I've tried adjusting the table cache and other variables that might seem related, but it hasn't helped. No helpful messages are logged in the error log.
Using a snapshot of the exact same code, compiled on Linux, I never see that problem. Ever.
My guess is that the problem is related to load, but I cannot reliably reproduce it (yet?).
Other than those two problems, it seems to work quite well. But I've only been running it for a few months with alpha and beta versions of MySQL. I'm fairly sure that all the other odd problems I've had with them are not related to LinuxThreads, but I can't yet say so with 100% certainty.
From what I've seen so far, using LinuxThreads ought to help even on single-CPU FreeBSD boxes.
If you are in the majority of FreeBSD users who never see major problems with MySQL on FreeBSD, there are some real advantages to staying there--not moving to Linux. The single biggest advantage is FreeBSD's VM subsystem. Under the workloads I've tested, it always outperforms Linux 2.4.x (where x is 9, 12, 16, and 18). When Linux decides to swap out data that is destined to be used again soon (think key buffer), FreeBSD doesn't go down that route. See my earlier blog entry for more details.
Linux doesn't have the problems that FreeBSD does. Threading works quite well. On the other hand, its VM can be a bit dumb at times. Turning off swap makes that non-issue if it does become a real problem.
Linux is still my preferred choice.
If anyone is able to assist in further debugging the FreeBSD problems--especially the LinuxThreads problems, I'd love the help.
I like Kasia's latest idea. Let's all try that. But not on the same day. Maybe we should all use our birthdays as the basis. Since I was born on June 4th, I'll be stupid on the 4th of every month.
Heh. I fear that it would get old after a few months. Maybe. But it'd be fun while it lasted, and it might even make a point.
I forgot to write this up yesterday and didn't feel like forging the blog entry date. Anyway, while taking a break from writing the book, I tried to finish up the work on friends.zawodny.com before I ship it off to get racked at WCNet where family.zawodny.com (which hosts jeremy.zawodny.com) has sat for about two years.
The new machine has 2 80GB Maxtor disks, one on each IDE channel. I have four RAID-1 filesystems (using Linux software RAID) on the drives: /, /usr, /var, and /home. I did some tinkering around and, after a few Goggle searches, I managed to get it to boot from either disk in the event of a failure. When hda failed (I unplugged it), the system booted from hdc just fine. And when hdc failed, it booted just fine from hda and ran in "degraded" mode until I rebooted with both disks and used raidhotadd to re-add the previously failed partitions to each RAID volume.
This is excellent. I had trouble getting this to work on machines in the past and opted for simply having the data on RAID volumes but no fail-over for booting. Now that I've seen it work, I see no need to ever really bother with hardware RAID. The built-in stuff works great.
I should note that I had tried to out-smart things (using the information like that found here) but it never worked for me. Once I tried to just follow the directions and told LILO that I wanted to use /dev/md0 rather than /dev/hda, it Just Worked. Excellent!
Combine all that with the fact that the Debian Woody install CD allows me to select from ext2, ext3, and ReiserFS at installation time, it's now far easier to build a robust Linux server that it used to be.
Strangely, I managed to get up before 9am (on a Sunday!) without the aid of an alarm clock. Not sure how that happened. I had some breakfast (muffin with jelly and peanut butter, and strawberry drink) while catching up on e-mail (not much) and blogs (a little more).
I had NPR on for background noise and because I often listen to Weekend Edition on KQED FM.
While listening, I got to hear a great discussion with Peter Gabriel about his new album: Up. I really enjoyed the piece and was thrilled to hear that his first studio album in 10 years is now available. I've been a big fan for a long, long time. I've already ordered it and hope share my thoughts on it later this week.
Sometimes the folks at K5 really come up with an excellent post:
You're passionate about something. It doesn't matter what you're passionate about. It could be basketweaving, jai lai, or how to make a convincing escargot out of common slugs. Whatever it is, it's something no one else could ever, ever, possibly give a damn about. But that won't stop you, no sir! You're going to tell the world about it and write...
Heh.
It seems that my body runs on a 25-26 hour clock, usually tending closer to 26 hours. This really drives me nuts sometimes. Scheduling is a real pain in the ass at times. Like when I'm not even close to falling asleep until 3am but I know I have to be somewhere at 10am. Because I've also noticed that I need a minimum of 6.5 hours of sleep each night. Otherwise the next day really sucks. I'm just dragging all day long.
Grr.
Maybe if I lived REALLY FAR NORTH (think Arctic Circle), I'd be better equipped to deal with the odd light/dark times.
Or not.
As many of you have noticed, the feeds are off now. Will they be back on? I hope so. Soon? I'm not sure.
What I will (and can) say is that the experience was very valuable for us.
There's been a lot of talk in various weblogs about Google News recently. But it has been around for a while. Apparently people didn't notice it until Google made the links more prominent. Here's a hint, keep an eye on Google Labs to see what they're playing with.
Anyway, a lot of folk are speculating on if and how Google will make money from their news service. Over at Loosely Coupled, you get a sampling of it.
This shouldn't surprise anyone, but there has been a lot of talk at Yahoo about Google recently. In fact, some of us were looking at Google News a few months back--and starting to worry a bit.
Rest assured that Google will money off their news service. And it will get better. Just watch.
Remember when Microsoft and AOL were the only competition we talked about? Yeah. Those days are gone now.
From Freshmeat:
gnuPod is a collection of tools which allow you to use your iPod under Linux and other UNIX- like operating systems. HFS+ and FAT32 formatted iPods are supported.
Cool. It was only a matter of time. Now how do I convince Apple to give me an iPod? :-)
Matt describes something I've been experiencing a lot recently:
You know you read too many primary sources when you read an article on slashdot and think, "I read that yesterday."
Yeah, ever since I got into weblogs. I don't rely on Slashdot, LinuxToday, and other sites for my information (notice that I didn't say "news") anymore.
First of all, thanks for all the great feedback. I see that a lot of folks are pulling it now. I'm working on some stats. It'll be interesting to see which stocks bloggers tend to watch, which aggregators they use, and so on.
I've seen a few folks make use of them in the weblogs already. postneo Matt picked up an Apple release, for example.
This is very cool. It's not often that I get such personal and quick feedback about something I worked on at Yahoo. Most of our stuff is so big that the feedback always gets filtered thru customer care and producers and so on. The upshot is that we engineers and hackers almost never see/hear/read it.
Keep it coming. :-)
A lot of you have requested features. The most popular request is to get the latest price of the stock as the first entry in the feed--linked to a quote page for more information. I like that idea a lot. I hope to implement that (and a few other things) in the next week.
The feedback is great. I can't stress that enough. We really don't get this for most of our "product launches".
If you mailed me personally or posted feedback and I haven't replied, don't worry. I've read it. I will reply. Thanks.
I think I'm on Dave's radar--at least for the moment. He not only picked up my announcement of the RSS feeds we're testing in Yahoo! Finance, he picked up on my rant from yesterday.
As I said in e-mail to Jon Udell the other day, Dave is the Slashdot of the blog community. Sort of. :-) You get linked from his site and you get lots of traffic. At least you'll never see some idiots posting "First Post!" and "Natalie Portman!" on Scripting News.
Dave, if you're reading this: Hi. Hope to meet you in person someday.
I've added a new blog category: Yahoo. Why? Because occasionally I post stuff about work, so I might as well categorize. Plus, some private e-mail has convinced me that it might be a good idea. (You know who you are. I think.)
Now let's see if I can not be stupid about what I use it for.
So, I called the power company promptly at 8am to find out what they were smoking. The allowed me to pay by credit card so that I could get reactivated today (and pay a $40 "reconnection fee" for the privilege).
I asked if they'd keep the information on file and use it from now on so that I didn't have to depend on the the postal system. No, they won't. I can call them every month and tell them my CC number and they'll gladly bill it. But they won't keep my information and save me the hassle.
I get the feeling that they were put on this plant to ensure that I have hassles to deal with.
My power was on 1 hour later.
So now I can be pissed at them with the lights on, at least. I have to find the time to bitch to the PUC about them. And remember to fill out the form that lets them take money directly from my checking account. In the meantime, I need to figure out why the payment never arrived. Luckily, I know the folks behind the scenes at Yahoo! BillPay.
Ask thinks so. More than a few times, I've heard both speculation and suggestions that someone should just buy what's left of VA Linux^H^H^H^H^HSoftware. "Someone" generally means IBM or Sun.
VA might make sense as part of a larger organization, but as it stands now things are very shaky. I remember a month or two ago, hearing that they just had another round of layoffs. That was something like the 5th or 6th.
Heck, VA is worth about $44 million today. They have at least that much in assets, So the real cost of buying that company would be very low--virtually nothing for a company like IBM, HPaq, or Sun.
The real question, of course, is what does buying VA really do? What the real product or knowledge that's left? Most of the folks who were VA are long gone.
No, that's not a typo. Just to make sure you read it the first time, I'll repeat it here.
Fuck Silicon Valley Power and Santa Clara Municipal Utilities
Yes, fuck them. Hard. No lubricant.
(Can you tell? I'm a little pissed right now.)
I came home tonight to find that my power was off and my UPSes were beeping at me. Puzzled, I checked my neighbors. They all had power. So I figured my breakers had tripped for some reason. I grabbed a flashlight and went to investigate. They were fine.
I sat down and ate my dinner (Chinese to go) by candle light. Of course, I had to find the candles first. Then I grabbed the phone book to call my local power monopoly. I explained the problem and the conversation went something like this.
Me: Hi. I just came home and found that my power is off.
Moron: Did you pay your bill?
Me: Yes, I'm pretty sure I did. But I can't tell you for sure. I use an on-line bill payment service and have for about a year and a half. And, well, my power is off--as you know. (I've since confirmed with BillPay that the payment went out as normal.)
Moron: What's your address.
(I tell him my address and he puts me on hold.)
Moron: Your service was turned off for non-payment.
Me: (Quite surprised.) What?! Well, can you turn it back on? I certainly have the money to pay whatever it is you think that I owe.
Moron: No, you'll need to take this up with billing in the morning.
Me: I called for emergency service and you're refusing to re-activate my power? Would you like to pay for the $100 of food that's going to spoil in my refrigerator?
Moron: Sir, you'll need to talk to billing at 8am tomorrow. (It was 7:30pm when I called.)
Me: Can you at least tell me when my power was shut off? I want to know if I should risk opening the refrigerator.
Moron: I have no idea.
Needless to say, I'm very pissed off. According to my records, they got their payment on-time. Did I get a threatening letter in the mail?
No.
Did I get a threatening phone call?
No.
Where do these assholes get off doing this?
I ended up driving up to Fry's to buy a 50 foot extension cord. I plugged it into the shared laundry room and used to power my refrigerator, one light, my DSL modem, and a laptop.
Grrrrrrrrr.
<rant>
I'm really starting to wonder why I bother living here at all. I pay $1,400 a month (twice the monthly payments on my old house) and for what? To deal with really bad traffic, moronic utility companies, and countless other bullshit.
Oh, that's right. I came out here to work at a company that has since forgotten how to innovate and take risks. Yippie.
Maybe I should start looking for a job somewhere that I can actually afford to live. It's not like I get paid enough to make up for the fact that a house here would cost me at least $500,000. Not even close.
At least somewhere else I'd have some room and maybe more of a life outside of work.
</rant>
Got a stock ticker for which you'd like to have an RSS news feed? Help test the beta RSS feeds we've put up o Yahoo Finance. Take your favorite ticker, say YHOO, and put this URL in your news aggregator:
http://rss.finance.yahoo.com/rss/get?ticker=YHOO
The pattern is pretty simple.
This hasn't been tested much yet, so I'd appreciate some feedback. There will likely be some outages as I tweak and adjust things. Also, if it works (or not) for you, it'd be good to know which aggregator you use. So far I know that NetNewsWire and AmphetaDesk like it.
UPDATE: I've fixed the missing language tag. Can a Radio user confirm that it's working now?
I dunno if I buy it yet. Just today there were a ton of stories on Linux Today about desktop Linux in one form or another:
What's behind this? Why is it such a hot topic this time? Red Hat 8.0? Sun? Lindows? Wal*Mart? I must be missing something. It's not adding up for me yet.
Dan thinks it is evolution. To me it's stupidity, laziness, and apathy. I have to respectfully disagree. Or maybe I'm just being stubborn.
For whatever reason, whenever I get e-mail from someone who:
I automatically assume that he or she is a little too dumb, a little too "pre-teen in a sex chartroom", a little too hip, a little too... annoying.
I relax my standards a bit for IM conversations, but even then overuse will get on my nerves pretty fast.
It appears that hfb has an evil toilet. It's a good thing that Josh never met it. She also found some amusing lines to use at work. I could use a few of those myself.
It's not like we didn't already know, but Dan's a prick. At least he's trying to justify it now. :-)
If you need to build some computers (like rack-mount servers) and live in the Bay Area, I recommend ASA Computers. I've business with them several times. Ask for Sean or Mike. Tell 'em I sent you. If you want to order stuff on-line, I highly recommend newegg.
From the fuzzy one, there's some PHP to query the Google API.
I visited a Sequoia backed start-up today to talk about MySQL. There's a ton of interest in it. Last week, I visited a company up in Pleasanton.
Speaking of MySQL, I got to hang out with David (one of MySQL's founders) and Marten (their CEO) for a couple hours yesterday.
Speaking of MySQL again, we've got a MySQL/FreeBSD box that appears to be on some serious crack. I really need to write up the "FreeBSD or Linux for MySQL?" document I've been mulling over in my head for a long time now.
There's an update to AmphetaOutlines available for all you AmphetaDesk users. Speaking of AmphetaDesk, I've been told that there's some really cool AmphetaOutlines-inspired stuff coming in the next version or so. Awesome.
No, I haven't gotten around to fixing the Yahoo Directory for Weblogs yet. A lot of folks have asked. I'm not ignoring anyone. I've just been busy.
Russell hasn't vanished, which is good. I'm interested in hearing his experience with Atkins. I've read the book and am interested. But I'm not yet sure what to make of it.
Ask reinstalled OS X and is happy now. I've been having a few odd problems with the TiBook ever since loading the Jaguar upgrade. Maybe I should reinstall too. Hmm.
Apparently jwz is getting old.
I almost posted this as a comment in response to Ask's comment about Sysadmin work (which can be fun or annoying as hell), but since I'm in the "write a long ass blog entry" mood...
For hire? Gee, Ask... Word on the street is that you've already got your hands full at work? :-)
Actually, I just like that line. "At first you might not think the price is right." That says so many things. I should start using that. It occurs to me that I'm always "for hire" to some degree or another. Adam Goodman (publisher of Linux Mag) likes to refer to me as a free agent. He's partly right. Just don't tell Uncle Sam.
Ah, I just posted it anyway. So call me redundant.
Do you grok mod_perl? Are you really good? Do you want to move to California? Let me know. Seriously. It seems that all the really good mod_perl folks are employed already.
Yes, syndication does matter. We're going to do something about this at Yahoo very soon. Watch this space. I'll need some beta testers.
There. Once nice, big, fuzzy (ask Kasia), warm blog entry. I can get on to other things now--like building that new rack-mount server. I'll probably name it friends.zawodny.com since it'll be sitting right near family.zawodny.com in Ohio. And the new scratchpost.zawodny.com is soon to join litterbox.zawodny.com on the west cost. Heh. I have my own little distributed server farm now. Anyone need hosting? Backup DNS? Other crap? I suddenly have a lot of spare capacity. I'll post specs later.
What a link-fest. Whew.
Just for kicks, I installed Debian Woody on a Sun LX50. It worked just fine, and I expected it to. The Cobalt group within Sun really has used commodity Intel server hardware to assemble this beast. And they've done a very nice job.
Once I'm done playing, I can put the Sun Linux restore CD in and bring the system back to "factory defaults" with Sun Linux installed once again. The whole process takes maybe 20 minutes if you're attentive to changing CDs (there are three of 'em).
Oh, cool. According to Radio Free Blogistan, there's a weblog category in the Yahoo directory.
And wouldn't ya know it? Mine isn't listed. Yet. I happen to know how to add items to the directory. Perhaps I'll remedy that minor oversight. :-)
Is your blog missing? Let me know. Let's see how large we can make that category!
If you happen to be using AmhetaDesk from CVS and run into an odd error about something not being an ARRAY reference in Channels.pm, apply this patch:
--- Channels.pm.old 2002-06-20 17:09:59.000000000 -0700
+++ Channels.pm 2002-09-14 18:29:03.000000000 -0700
@@ -130,8 +130,13 @@
# templates. this matches the rss 1.0 layout.
my $items_copy; $items_copy->{item} = $data->{channel}->{item};
undef $data->{channel}->{item}; # remove this hierarchy.
- foreach my $item (@{$items_copy->{item}}) {
- push(@{$data->{item}}, $item);
+ if (ref $items_copy->{item} ne 'ARRAY') {
+ # uh oh!
+ }
+ else {
+ foreach my $item (@{$items_copy->{item}}) {
+ push(@{$data->{item}}, $item);
+ }
}
}
elsif ($channel_xml =~ //i) {
It fixed it for me. Patch is on its way to the AmphetaDesk hackers, too.
Yeay for Open Source.
Update: Morbus Iff (the creator of AmphetaDesk) tells me that it's already fixed in CVS. Woohoo!
After a few month of lurking and leaving odd, sometimes offensive comments in the blogs of my friends, Dan Isaacs now has one. He's calling it Because I can. He still has some tweaking and customization to do, but there's content there already.
So, if you're one of the many that have been harassed by Dan, now's your change to return the... uh... favor. :-)
Okay, I've just finished greatly simplifying the CSS for my blog. I've even commented the CSS so I'll remember what controls what. It seems to look a bit nicer on the browsers I've tried. But my TiBook is currently loaned out, so I'd appreciate feedback from Mac (and other) users. Tell me if you like it or not. If not, why? What would you change? And please tell me which browser you use.
I should note that it's completely unreadable in Netscape 4. But I no longer care. If you're still using that old, steaming pile of crap, not being able to read my weblog is the least of your problems. Just disable style sheets and it'll be just fine. Really. I tried.
Hm. It looks like Simon wants to integrate his mail, news, and rss/weblog reading into a single application. In his case, that application is gnus the killer Mail/News reader for Emacs. What he's saying really hits home for me. I recently wrote about my frustrations with RSS aggregators too.
I have to agree with Simon. Having an rss back-end for gnus would be excellent. However, I'd like to go a step further and propose something for those GUI inclined folks (not necessarily me, since I use mutt to read mail): Mozilla needs RSS aggregation capabilities.
Think about it. All the pieces are there. Mozilla is infinitely scriptable. It certainly knows how to fetch content via HTTP. It knows how to parse XML. Building a UI with Mozilla isn't terribly difficult.
Is anyone working on this. If not, why? This seems like a killer app. Put Mail, News, and Weblogs on equal footing in the only Open Source cross-platform platform (that's not a typo) we have.
I ran a quick google search to see who's working with RSS and Mozilla. I found a few interesting items:
And there's some RDF info on mozilla.org too.
But as far as I can tell, nobody has done this. Hmm. Someone really should.
Starting about 1 year ago, I get catalogs every few months from The Territory Ahead. I had never heard of them before. The appear to be an L.L. Bean for people who really have cash to burn. The stuff is expensive.
I'll probably never buy from them, and I'm really not sure how or why they got my name. But for some reason I flip thru the catalog each time it shows up. Then I throw it away.
Maybe L.L. Bean sold my info. I'm not sure.
I should take a moment to mention that I'm a bit of an L.L. Bean fan. In fact, I wrote about them before (in my old on-line journal, from before I had ever heard of a "weblog").
Anyway, I thought I'd share. Maybe I'm just recovering from blog-withdrawl or something.
Though we've had a subscriptions page up for a while on the PHP Journal web site, there were two problems with it.
Those are now fixed. If you subscribed before, please head over to the PHP Journal site and re-subscribe. If you haven't yet... Well, what are you waiting for?
If you like reading about Mac OS X topics and looking at beautiful women, look no farther than the inluminent weblog. Then again, if that's not your style, don't go there. :-)
It looks like Phil Windley has an iBook to play with. I see that he's identified some of the same strengths and weaknesses that I noted. (Apparently we have similar ThinkPads and would really like a high-res LCD on a Mac notebook.)
It'll be interesting to hear how well it integrates with the infrastructure in Utah's government.
In other news, I'm TiBook-less for a while. Jeffrey Friedl and his wife have mine. They're thinking of getting an iBook and wanted to play with something similar for a couple weeks before taking the plunge. I'm pretty sure they'll end up doing it, considering that he just bought a WAP11 wireless access point so that they could experience wireless computing. :-)
So what have I been using at home? My ThinkPad T23 with XP? No, my 3-year old Thinkpad 600E w/Debian Linux. Hey, I've got xterms, Emacs, and Galeon. What else do I really need?
Wow, it's been a busy week. I was totally swamped for several days dealing with the remember.yahoo.com MySQL servers and related stuff. And then I used a day or two to recover (sleep, shower, etc).
Anyway, I made some interesting discoveries along the way. The most surprising one had to do with thread caching on Linux when you have a busy MySQL server--busy in a particular way, mind you.
You see, we had a single master server which all the web servers could connect to (using PHP) whenever someone made a change. That includes creating a tile (there were several hundred thousand tiles created), approves a tile, marks one as "cool", and so on. All told, the master was quite busy.
Because there were between 20 and 45 front-end web servers during that time, and each could have had up to 70 apache processes that might have needed to connect, we faced a problem. That meant that the master needed to handle up to 3,150 connections in the worst case (that's 45 x 70). Most of the PHP code used mysql_pconnect() to hold persistent connections.
Rather than worry about how to do that, I made sure that the wait_timeout was set to a very low value: 15 seconds. That means MySQL would close any connection that was idle for more than 15 seconds. But I didn't realize the extent of the problem until I started getting reports from the web servers that the master was refusing connections. Why? Because I had set the maximum number of connections to a reasonable value in the master's my.cnf file:
set-variable = max_connections=180 set-variable = max_user_connections=140
And at that time, the wait_timeout was set to 600 seconds (10 minutes). Clearly that was a problem. There were a lot of idle clients holding connections and blocking out new clients from connecting and getting real work done.
What to do?
We could have stopped using mysql_pconnect(), but as you'll see, that wouldn't have solved the underlying problem.
I needed to adjust the settings. But I wasn't sure what values to use. And I really didn't want to keep stopping and starting the master. That would just suck. Then I remembered that we were running MySQL 4.0.4. I'd has a new feature that allows you to change most of the server settings on on the fly without restarting! Read about here, it in the on-line manual.
Excellent!
All I needed to do was execute a few variations on this command:
SET GLOBAL wait_timeout=60;
(with different values in the place of "60") to try and strike a balance between letting new clients in and kicking out already connected users too quickly.
Ultimately, I settled on a timeout of 15 seconds.
But that had an interesting and unanticipated side-effect. It meant that the Linux server was having to create new threads (MySQL is a multi-threaded server) at a very high rate. That sucks up a measurable amount of CPU time.
How much CPU time? By the time I got around to looking at the output of SHOW STATUS and seeing this:
| Threads_cached | 0 | | Threads_created | 270194 | | Threads_connected | 46 | | Threads_running | 28 |
Things were pretty bad. The machine had very little idle CPU time--probably 5-10% at the most. But it really wasn't doing that much work--maybe 40 queries per second. I was a bit puzzled. But that Threads_created number jumped out at me. It was high and increasing rapidly.
Luckily I remembered the thread_cache setting. So I decided to investigate (using the new syntax for examining server variables):
mysql> SELECT @@global.thread_cache_size; +---------------------+ | @@thread_cache_size | +---------------------+ | 0 | +---------------------+ 1 row in set (0.00 sec)
Uh oh. I never set the thread cache in my.cnf, so it has assumed the default. That's bad. It's like removing the pre-forking capabilities of Apache 1.3 and letting it get pounded on a busy web site. The "fork a new process for each new request" gets pretty expensive pretty quickly.
Ugh!
Luckily the thread cache is also tunable on the fly now. So all I had to do was this:
SET GLOBAL thread_cache_size=40;
I took a guess and figured that by caching 40 threads, we'd be saving a lot of work. And boy was I right!
In the other window, where I was running vmstat 1 I noticed a dramatic change. The idle CPU on the machine immediately went from 5-10% to 35-40%
If only I had thought of that sooner!
So the moral of the story is this: If you have a busy server that's getting a lot of quick connections, set your thread cache high enough that the Threads_created value in SHOW STATUS stops increasing. Your CPU will thank you.
I don't feel bad though. We were all going nuts to try and tune/optimize the code and servers while it was running and had very little sleep. Thread caching really wasn't the worst of our problems. But it became the worst after we had fixed all the bigger ones.
It was quite a learning experience.
So I'm up just before 5:30am. Why? David Filo paged me because some database servers was pegged on CPU. I investigate. I find a couple of horribly un-optimized queries. They were doing full table scans due to lack of indexes. Even though the tables are essentially in RAM, that really CPU intensive, 'cause all the rows needed to examined.
The moral of the story: Optimize your SQL before running it on my servers and telling the whole world to look at it.
Grr. I guess I'm gonna be up for a while.
It's up. Visit remember.yahoo.com, which is powered by Apache, PHP, and MySQL. I spent a fair amount of today getting the MySQL servers ready for it.
After Justin asked for a slash-like site for news about SpamAssassin, I volunteered to get something together. It currently lives on news.spamassassin.zawodny.com, which will eventually answer to news.spamassassin.org as well. Now I need to get some feedback from the core SA folks and anyone else who wants to poke at it.
I have to agree with Kasia on this one. There are a lot of "programmers" out there who couldn't think their way out of a paper bag. And many are now complaining that they can't find jobs paying enough money.
I don't mean to discount the good, smart folks who are looking for work. I hope they're they first ones hired when things turn around. But as for the rest of the bunch, maybe Burger King has openings?
I make a decent living in my job. In fact, the only real reason that I'd like to make more money is that the cost of housing (not living--housing) is so damned high in the Bay Area. When I moved out here, I had to sell my $92,000 1 and 1/2 story house in Ohio. I way making $730/month house payments. I moved into a crappy apartment in Santa Clara (where I still live) with a $1,400/month rent. That's right. I had to pay twice as much for 1/2 the space. I have no yard, no garage, no patio, and no view.
Ugh. Do I think I should make more money because I'm a programming god of some sort? No. But I'd really like to live in a house (or at least a townhouse) again. The problem is that every decent house costs a minimum of $700,000 and event a decent townhouse goes for over $350,000. At this rate, I'm gonna be stuck in my crappy apartment for a long, long time. Or I'll just move.
I highly recommend reading this article over on Y! News, an Op/Ed piece from Ted Rall.
It starts out with:
Are you the kind of person who believes that attaching a plastic American flag to your SUV makes a major patriotic statement? If so, you're no doubt anticipating the looming anniversary of the September 11 attacks with the enthusiasm of a nine-year-old on Christmas morning.
Then there's the rest of us.
And just gets better from there.
I have this theory. Non-Drowsy medicine means "doesn't really work worth a damn." So far, I've yet to be proven wrong. The only allergy medicines that really help me also sorta knock me out. And now, the first cold medicine I've tried today (Multi-Symptom Tylenol Cold) appears to be useless. I still have a runny nose and sore throat. Well, I guess it's time to try the Benadryl Allergy/Cold and see what it does (other than try to make me sleep).
Grr.
I woke up this morning with what appears to be a cold. That really sucks. I mean, a cold at this time of year. It's in the 70s and 80s outside. WTF?!
Grr.
Check out these graphs that chart the demand for Java and C++ programmers since September 1999. Then look at the Nasdaq chart on that page. (As seen on Russell Beattie's Notebook)
I was very interested in this story over on K5. Not just because it's "interesting" but because it's the sort of story that makes K5 a unique and valuable community. It's really amazing to see folks come together and discuss a whole range of topics and ideas on a daily basis, and then to realize that most of these people really don't know who they're talking to.
On a more personal note, the idea of being a "Geek Volunteer" in a less developed country sounds like something I might want to try someday. Not today or tomorrow, but I can certainly see myself doing it at some point. I really think I'd end up learning far more than I could teach anyone. It would be a great experience, I'm sure. The real question is how to decide when it's right to extract yourself from day-to-dat living and do something like this.
Maybe you "just know" when the time is right.
Anway, go read the story. I look forward to reading the author's progress.
I just saw an Oracle banner ad on Slashdot that read: "Looking for the #1 database on Linux?" Curious, I clicked on it. After several redirects, it took me to this page.
Just for the record, it says, among other things:
Oracle9i Database on Linux is a fast, reliable and cost-effective way to grow your business without growing your budget.
Nowhere on that page do they provide justification for their claim that Oracle9i is the "#1 database on Linux." You know why? Because it's not. Not by a long shot. You see, Oracle is expensive. Most Linux users are quite happy with PostgreSQL or MySQL.
"Ah," you say, "they're marketing to the corporate world." Maybe. But on Slashdot?!?! I can think of about 50 other sites that they ought advertise on first.
Oh, and if Oracle is so good, why has Slashdot's parent company decided to use IBM's DB2 as the back-end for their only product?
That page also says...
Voted Linux Journal Editors' Choice for "sheer performance", Oracle9i Database on Linux is Unbreakable.
What the hell to Linux Journal's editors know about databases? Are any of them DBAs? What sort of applications have they been building with Oracle?
Oh, well. Back to my real writing.
Update: Google speaks the truth. When you search Google for "The #1 Database on Linux" the first link you get is for MySQL. Oracle is #3 on the list. Heh. Nice try Mr. Ellison. (For some reason Linux Journal is #4 in the results. Must have a lot more to do with "linux" than "database".
I really wish I could write English text at even half the speed that I code at. In the last two hours, I've written about two pages of text. Ugh.
I should have told everyone that the book would be done in 2005 so I'd have a chance of getting it done early.
Woohoo. My blog is in the Fuzzyblog's FontSafe category rather than the FontBitch category. I'm glad someone is beating this drum.
We have this argument at work all the time, and I refuse to put fixed-sized font specs in the CSS I build. If you look at the in-line style sheet here, you'll notice that it's all relative. Just the way it ought to be.