9

A Better diff Or What To Do When GNU diff Runs Out Of Memory ("diff: memory exhausted")


Posted by Artem Russakovskii on May 12th, 2008 in Databases, Linux, Programming

Updated: June 1st, 2008

Recently I ran into major problems using GNU diff. It would crash with "diff: memory exhausted" after only a few minutes trying to process the differences between a couple 4.5GB files. Even a beefy box with 9GB of RAM would run out of it in minutes.

There is a different solution, however, that is not dependent on file sizes. Enter rdiff – rsync's backbone. You can read about it here: http://en.wikipedia.org/wiki/Rsync (search for rdiff).

The upsides of rdiff are:

  • with the same 4.5GB files, rdiff only ate about 66MB of RAM and scaled very well. It never crashed to date.
  • it is also MUCH faster than diff.
  • rdiff itself combines both diff and patch capabilities, so you can create deltas

    • …

    Read the rest of this article »