After about three weeks of fighting with an optimization problem at work (including an interesting detour into Perl guts), we have finally prevailed. The processing didn't run out of memory and die after the first few iterations.
The solution was to re-think the algorithm yet again. Our first few attempts helped but didn't get us far enough. The most recent change, however, has kept us far enough under the memory limit to ensure that it should run to completion. I'll know for sure in the morning.
It'll be interesting to see what happens when the Perl code is converted to Java. Will it be faster or slower? Use more or less memory? How many more or fewer lines of code will it require?
Time will tell.
Posted by jzawodn at March 03, 2003 07:43 PM