Programmers can be so damned stupid sometimes.
Take me for example.
I've been working to optimize and adjust some code at work. I can't tell you what it does but I can tell you that it's too slow and uses too much memory. It's Perl. I know Perl. I'd like to think I'd know it pretty well, having used it for around nine years now.
In tracking down this memory problem, I've learned a lot about what a memory pig Perl can be. But that's a topic for another blog entry. The reall issue is how I've been tracking the problem. I'd get a hunch that the %foo hash was way too big and causing the process to die. So I'd convert it to a tied hash backed by Berkeley DB. And I'd run it again. It would again die.
Of course, this never happens in my small and quick to test data. It only happens with the full load (between 6 and 17 million, uhm, phrases). And it takes anywhere from 35 to 60 minutes for it to die. So you can guess how productive this makes me with an average 45 minute test cycle.
I've finally decided to just resort to a classic debugging technique: the binary search. Well, with a twist. Thanks to Ray, I'm using Devel::Size to periodically dump the memory use (or some approximation of it--that's another story) out to a log.
Why I didn't start this a few days ago is beyond me.
No, wait. It's not. It's because every time I tried something new, followed a new hunch, I was convinced that it was the solution.
Someone slap me next time I do this.
Posted by jzawodn at February 20, 2003 09:29 PM