After about three weeks of fighting with an optimization problem at work (including an interesting detour into Perl guts), we have finally prevailed. The processing didn't run out of memory and die after the first few iterations.
The solution was to re-think the algorithm yet again. Our first few attempts helped but didn't get us far enough. The most recent change, however, has kept us far enough under the memory limit to ensure that it should run to completion. I'll know for sure in the morning.
It'll be interesting to see what happens when the Perl code is converted to Java. Will it be faster or slower? Use more or less memory? How many more or fewer lines of code will it require?
Time will tell.
Posted by jzawodn at March 03, 2003 07:43 PM
After all the discussion in the previous posting, it would be interesting to hear more about the refactoring. If you're at liberty to, of course.
I, for one, would love to hear what the end results were on the Java rewrite. (And I'm still tempted to write a lower-memory-usage hash implementation for those cases where you only really need an integer, rather than a full perl scalar attached, but that's a separate thing)
Well, in general, expect it to be slower and take more memory for hashes. Then again, that test doesn't try hashes as large as those you're using.
And judging by the usual ratio of 1 : 10000 sloc for mapping from perl into almost any other imperative language, expect to write more code, too. (Okay, maybe that's a bit of an exaggeration...)
