| Share |
Updated: June 1st, 2008
Recently I ran into major problems using GNU diff. It would crash with "diff: memory exhausted" after only a few minutes trying to process the differences between a couple 4.5GB files. Even a beefy box with 9GB of RAM would run out of it in minutes.
There is a different solution, however, that is not dependent on file sizes. Enter rdiff – rsync's backbone. You can read about it here: http://en.wikipedia.org/wiki/Rsync (search for rdiff).
The upsides of rdiff are:
- with the same 4.5GB files, rdiff only ate about 66MB of RAM and scaled very well. It never crashed to date.
- it is also MUCH faster than diff.
- rdiff itself combines both diff and patch capabilities, so you can create deltas and apply them using the same program
The downsides of rdiff are:
- it's not part of standard Linux/UNIX distribution – you have to install the librsync package.
- delta files rdiff produces have a slightly different format than diff's.
- delta files are slightly larger (but not significantly enough to care).
- a slightly different approach is used when generating a delta with rdiff, which is both good and bad – 2 steps are required. The first one produces a special signature file. In the second step, a delta is created using another rdiff call (all shown below). While the 2-step process may seem annoying, it has the benefits of providing faster deltas than when using diff. In fact, you can pipe the first step into the second one without any trouble if you want, which is what I ended up doing).
Usage:
1 2 3 4 5 6 7 8 9 | $ rdiff signature ORIGINAL.txt SIGNATURE.sig $ l -h SIGNATURE.sig -rw-r--r-- 1 user users 25M 2008-04-23 22:32 SIGNATURE.sig $ rdiff delta SIGNATURE.sig MODIFIED.txt DELTA.rdiff $ l -h DELTA.rdiff -rw-r--r-- 1 user users 82M 2008-04-23 22:36 DELTA.rdiff |
And here's what you would do to reassemble MODIFIED.txt:
1 2 3 4 5 6 | $ rdiff patch ORIGINAL.txt DELTA.rdiff MODIFIED_REASSEMBLED.txt $ l *.txt -rw-r--r-- 1 user users 4,471,493,588 2008-04-23 20:24 MODIFIED.txt -rw-r--r-- 1 user users 4,471,493,588 2008-04-23 22:44 MODIFIED_REASSEMBLED.txt -rw-r--r-- 1 user users 4,403,302,981 2008-04-23 20:20 ORIGINAL.txt |
Just as expected – everything matches.
Now, all of this could have been done in one go like this:
1 | rdiff signature ORIGINAL.txt | rdiff delta -- - MODIFIED.txt DELTA.rdiff |
As far as my usage of such a useful diff program, I was doing CSV dumps of certain fields from a MySQL database, like so:
1 | SELECT * FROM table WHERE some_condition='1' ORDER BY id DESC INTO OUTFILE '/home/dump/dump.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'; |
and then applying rdiff to get the [quite small] daily deltas.
That's all folks!
Artem Russakovskii is a San Francisco programmer, blogger, and future millionaire (that last part is in the works). Follow Artem on Twitter (@ArtemR) or subscribe to the RSS feed.
In the meantime, if you found this article useful, feel free to buy me a cup of coffee below.
beer planet is a blog about technology, programming, computers, and geek life. It is run by Artem Russakovskii - a local San Francisco geek who currently works at 
Artem loves commenting on other people's blogs and reading comments on his own, so go ahead and leave one.
Thank you for commenting on the continuation of the Lazyest Gallery project.
Just a nitpick: rdiff and rsync don't share any code. rdiff's backend, librsync, just happens to be another implementation of the rsync running-checksum algorithm.
The delta files generated by "rsync –write-batch" are also very different to the ones rdiff produces.
@nailbiter
Thanks for the correction – I just assumed that rsync and librsync are related for obvious reasons.
Does anyone know how to view the Delta output "diff"-style? or any other style?
[...] tagged gnuOwn a Wordpress blog? Make monetization easier with the WP Affiliate Pro plugin. A Better diff Or What To Do When GNU diff Runs Out… saved by 2 others kidamazone bookmarked on 05/17/08 | [...]