There is a different solution, however, that is not dependent on file sizes. Enter rdiff - rsync's backbone. You can read about it here: http://en.wikipedia.org/wiki/Rsync (search for rdiff).

The upsides of rdiff are:

• with the same 4.5GB files, rdiff only ate about 66MB of RAM and scaled very well. It never crashed to date.
• it is also MUCH faster than diff.
• rdiff itself combines both diff and patch capabilities, so you can create deltas and apply them using the same program

The downsides of rdiff are:

• it's not part of standard Linux/UNIX distribution - you have to install the librsync package.
• delta files rdiff produces have a slightly different format than diff's.
• delta files are slightly larger (but not significantly enough to care).
• a slightly different approach is used when generating a delta with rdiff, which is both good and bad - 2 steps are required. The first one produces a special signature file. In the second step, a delta is created using another rdiff call (all shown below). While the 2-step process may seem annoying, it has the benefits of providing faster deltas than when using diff. In fact, you can pipe the first step into the second one without any trouble if you want, which is what I ended up doing).

Usage:

1
2
3
4
5
6
7
8
9

$ rdiff signature ORIGINAL.txt SIGNATURE.sig
 
$ l -h SIGNATURE.sig
-rw-r--r-- 1 archon810 users 25M 2008-04-23 22:32 SIGNATURE.sig
 
$ rdiff delta SIGNATURE.sig MODIFIED.txt DELTA.rdiff
 
$ l -h DELTA.rdiff
-rw-r--r-- 1 archon810 users 82M 2008-04-23 22:36 DELTA.rdiff

And here's what you would do to reassemble MODIFIED.txt:

1
2
3
4
5
6

$ rdiff patch ORIGINAL.txt DELTA.rdiff MODIFIED_REASSEMBLED.txt
 
$ l *.txt 
-rw-r--r-- 1 archon810 users 4,471,493,588 2008-04-23 20:24 MODIFIED.txt
-rw-r--r-- 1 archon810 users 4,471,493,588 2008-04-23 22:44 MODIFIED_REASSEMBLED.txt
-rw-r--r-- 1 archon810 users 4,403,302,981 2008-04-23 20:20 ORIGINAL.txt

Just as expected - everything matches.

Now, all of this could have been done in one go like this:

1

rdiff signature ORIGINAL.txt | rdiff delta -- - MODIFIED.txt DELTA.rdiff

As far as my usage of such a useful diff program, I was doing CSV dumps of certain fields from a MySQL database, like so:

1

SELECT * FROM table WHERE some_condition='1' ORDER BY id DESC INTO OUTFILE '/home/dump/dump.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"';

and then applying rdiff to get the [quite small] daily deltas.

That's all folks!

• Converting from CVS to SVN: Developer's Notes And Why SVN Is Better
• Unpublished Way To Get YouTube Search Results as Custom RSS Feeds
• Watch - A Useful Linux Command You May Have Never Heard Of
• Mass Renaming Directories And Files Using Total Commander
• Navicat For MySQL Bugs Filed

Check this out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

cpan[1]> install SOAP::Lite
CPAN: Storable loaded ok (v2.18)
Going to read /root/.cpan/Metadata
  Database was generated on Tue, 29 Apr 2008 18:29:45 GMT
CPAN: YAML loaded ok (v0.66)
Going to read /root/.cpan/build/
............................................................................DONE
Found 149 old builds, restored the state of 109
Warning: Cannot install SOAP::Lite, don't know what it is.
Try the command
 
    i /SOAP::Lite/
 
to find objects with matching identifiers.
CPAN: Time::HiRes loaded ok (v1.9713)

Huh? Okay…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

cpan[2]> i /SOAP::Lite/    
Module    ResourcePool::Command::SOAP::Lite::Call (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module    ResourcePool::Factory::SOAP::Lite (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module    ResourcePool::Resource::SOAP::Lite (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module    SOAP::Lite::Deserializer::XMLSchema1999 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::Deserializer::XMLSchema2001 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::Deserializer::XMLSchemaSOAP1_1 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::Deserializer::XMLSchemaSOAP1_2 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::InstanceExporter (SMEISNER/SOAP-Lite-InstanceExporter-0.02.tar.gz)
Module    SOAP::Lite::Packager   (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::Simple     (LLAP/SOAP-Lite-Simple-1.9.tar.gz)
Module    SOAP::Lite::Simple::DotNet (LLAP/SOAP-Lite-Simple-1.4.tar.gz)
Module    SOAP::Lite::Simple::Real (LLAP/SOAP-Lite-Simple-1.4.tar.gz)
Module    SOAP::Lite::Utility    (BRYCE/SOAP-Lite-Utility-0.01.tar.gz)
Module    SOAP::Lite::Utils      (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
14 items found

Wtf? Let's try something else.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

cpan[8]> i /SOAP.*Lite/                           
Distribution    BRYCE/SOAP-Lite-Utility-0.01.tar.gz
Distribution    BYRNE/SOAP/SOAP-Lite-0.60a.tar.gz
Distribution    DYACOB/SOAP-Lite-ActiveWorks-0.10.tar.gz
Distribution    DYACOB/SOAP-Lite-SmartProxy-0.11.tar.gz
Distribution    LLAP/SOAP-Lite-Simple-1.4.tar.gz
Distribution    LLAP/SOAP-Lite-Simple-1.9.tar.gz
Distribution    MKUTTER/SOAP-Lite-0.71.04.tar.gz
Distribution    MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz
Distribution    SMEISNER/SOAP-Lite-InstanceExporter-0.02.tar.gz
Module    Catalyst::Action::SOAP::DocumentLiteral (DRUOSO/Catalyst-Controller-SOAP-0.8.tar.gz)
Module    Catalyst::Action::SOAP::DocumentLiteralWrapped (DRUOSO/Catalyst-Controller-SOAP-0.8.tar.gz)
Module    Catalyst::Action::SOAP::RPCLiteral (DRUOSO/Catalyst-Controller-SOAP-0.8.tar.gz)
Module    Catalyst::Controller::SOAP::DocumentLiteralWrapped (DRUOSO/Catalyst-Controller-SOAP-0.8.tar.gz)
Module    Net::DRI::Transport::HTTP::SOAPLite (PMEVZEK/Net-DRI-0.85.tar.gz)
Module    ResourcePool::Command::SOAP::Lite::Call (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module    ResourcePool::Factory::SOAP::Lite (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module    ResourcePool::Resource::SOAP::Lite (MWS/ResourcePool-Resource-SOAP-Lite-1.0101.tar.gz)
Module  = SOAP::Lite::Deserializer::XMLSchema1999 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module  = SOAP::Lite::Deserializer::XMLSchema2001 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module  = SOAP::Lite::Deserializer::XMLSchemaSOAP1_1 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module  = SOAP::Lite::Deserializer::XMLSchemaSOAP1_2 (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::InstanceExporter (SMEISNER/SOAP-Lite-InstanceExporter-0.02.tar.gz)
Module  = SOAP::Lite::Packager   (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
Module    SOAP::Lite::Simple     (LLAP/SOAP-Lite-Simple-1.9.tar.gz)
Module    SOAP::Lite::Simple::DotNet (LLAP/SOAP-Lite-Simple-1.4.tar.gz)
Module    SOAP::Lite::Simple::Real (LLAP/SOAP-Lite-Simple-1.4.tar.gz)
Module    SOAP::Lite::Utility    (BRYCE/SOAP-Lite-Utility-0.01.tar.gz)
Module  = SOAP::Lite::Utils      (MKUTTER/SOAP-Lite-0.71.04.tar.gz)
28 items found

Aha! It's hiding under a Distribution. Tricky, tricky.

1
2

cpan
install MKUTTER/SOAP-Lite-0.71.04.tar.gz

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

  CPAN.pm: Going to build M/MK/MKUTTER/SOAP-Lite-0.71.04.tar.gz
 
We are about to install SOAP::Lite and for your convenience will provide
you with list of modules and prerequisites, so you'll be able to choose
only modules you need for your configuration.
 
XMLRPC::Lite, UDDI::Lite, and XML::Parser::Lite are included by default.
Installed transports can be used for both SOAP::Lite and XMLRPC::Lite.
 
Press  to see the detailed list.  
 
Feature                       Prerequisites                Install?
----------------------------- ---------------------------- --------
Core Package                  [*] Scalar::Util             always  
                              [*] Test::More                       
                              [*] URI                              
                              [*] MIME::Base64                     
                              [*] version                          
                              [*] XML::Parser (v2.23)              
Client HTTP support           [*] LWP::UserAgent           always  
Client HTTPS support          [*] Crypt::SSLeay            [ yes ] 
Client SMTP/sendmail support  [ ] MIME::Lite               [ no ]  
Client FTP support            [*] IO::File                 [ yes ] 
                              [*] Net::FTP                         
Standalone HTTP server        [*] HTTP::Daemon             [ yes ] 
Apache/mod_perl server        [ ] Apache                   [ no ]  
FastCGI server                [ ] FCGI                     [ no ]  
POP3 server                   [*] MIME::Parser             [ yes ] 
                              [*] Net::POP3                        
IO server                     [*] IO::File                 [ yes ] 
MQ transport support          [ ] MQSeries                 [ no ]  
JABBER transport support      [ ] Net::Jabber              [ no ]  
MIME messages                 [*] MIME::Parser             [ yes ] 
DIME messages                 [*] IO::Scalar (v2.105)      [ no ]  
                              [ ] DIME::Tools (v0.03)              
                              [ ] Data::UUID (v0.11)               
SSL Support for TCP Transport [ ] IO::Socket::SSL          [ no ]  
Compression support for HTTP  [*] Compress::Zlib           [ yes ] 
MIME interoperability w/ Axis [ ] MIME::Parser (v6.106)    [ no ]  
--- An asterix '[*]' indicates if the module is currently installed.
 
Do you want to proceed with this configuration? [yes] 
Checking if your kit is complete...
Looks good
Writing Makefile for SOAP::Lite
cp lib/SOAP/Packager.pm blib/lib/SOAP/Packager.pm
cp lib/XML/Parser/Lite.pm blib/lib/XML/Parser/Lite.pm
...
Writing /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/SOAP/Lite/.packlist
Appending installation info to /usr/lib/perl5/5.10.0/i686-linux/perllocal.pod
  MKUTTER/SOAP-Lite-0.71.04.tar.gz
  /usr/bin/make install  -- OK

The latest version of SOAP::Lite is installed, time to pat yourself on the back and write some code to actually use it.

• The Repositories For apt-get In SUSE 10.2
• cpan - The Perl Module Manager
• Parsing JSON In Perl By Example - SouthParkStudios.com South Park Episodes
• Do NOT Use This Perl Module: Passwd::Unix
• How To Install The Latest SOAP::Lite Using Perl CPAN

Virtual walking aside, there are some really creative uses of this feature posted in this video by the Google team today. I never myself thought to check my own street for street cleaning signs - saves a trip downstairs! Or look at the toll road prices (like the Bay bridge toll). Or at least watch people falling off their bikes. Anyway, just watch the video (thanks to zefrank for posting it).

• Cambridge, UK, Week 1
• Cambridge, UK, Week 2
• The Magic HD-DVD Key 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
• Google Maps New Feature: Visual Route Dragging
• Interesting Uses For Google Streetview (Video By Google)

The best solution I could find to this problem to date was posted at http://www.perlmonks.org/?node_id=19119 and involved opening a piped filehandle. It worked quite well but always felt like a hack (which it was). Having used the new Perl 5.10 for a few months, I was shocked today to find this new variable that I've been dreaming about for years:

1

${^CHILD_ERROR_NATIVE}

This variable gives the native status returned by the last pipe close, backtick command, successful call to wait() or waitpid(), or from the system() operator. See perlrun for details. (Contributed by Gisle Aas.)

http://search.cpan.org/dist/perl-5.10.0/pod/perl5100delta.pod#New_internal_variables

I've just tested it and it works as described. Finally!.. what else can I say?

• Watch - A Useful Linux Command You May Have Never Heard Of
• FTPRush Cleanup Script
• Quick Perl Snippet: Finding If A File Has A Media Extension Using Regex
• MySQL Conference Liveblogging: Monitoring Tools (Wednesday 5:15PM)
• How Do I Get Both The Return Value And Text In Perl? Backticks vs. System() (Perl 5.10)

1

tar -tzf file.tar.gz

-t: lists files
-f: instructs tar to deal with the following filename (file.tar.gz)
-z: informs tar that the it's dealing with a gzip file (-j if it's bzip2)

• Mass Renaming Directories And Files Using Total Commander
• Quick Perl Snippet: Finding If A File Has A Media Extension Using Regex
• How To Delete All Messages From A Folder In Pine
• Some Useful vim Commands - My vim Cheatsheet
• How To List Files Within tgz (tar.gz) Archives

Passwd::Unix will corrupt your /etc/shadow file and rearrange login names and their corresponding password hashes.

The current version of Passwd::Unix corrupted my /etc/shadow upon only
calling the passwd() function. Immediately users started to report not
being able to login.

After examining the situation, I found that Passwd::Unix rearranges all
users in /etc/shadow in some way, but it only does it to the
usernames, and not the password hashes. Thus, you will get corrupted accounts. Moreover,
users are now able to login to one OTHER account, not their own,
depending on how the usernames got shuffled.

Thankfully, I had a recent backup but I definitely don’t want anyone
else to suffer.

I’m using perl 5.10, SUSE 10.3. If it’s incompatible with SUSE, it needs
to say so and exit.

I've filed the bug here: http://rt.cpan.org/Public/Bug/Display.html?id=35323.

You have been warned.

• cpan - The Perl Module Manager
• FTPRush Cleanup Script
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• Do NOT Use This Perl Module: Passwd::Unix
• How To Install The Latest SOAP::Lite Using Perl CPAN

Last year, Sun already made an announcement of JavaFX, a Java-based mobile platform but didn't provide any concrete timelines, so I was excited to hear the more on the subject. With Apple iPhone's advent last year and Google entering the same space later on this year with Android, Sun's addition to the game definitely won't hurt consumers. After all, competition usually leads to better products.

So when is the phone coming, Jonathan? My guess is you're going to try as hard as possible to compete with Google's second half of 2008 timeline but are you going to manage to beat it? And will it blow us away? I guess time will show.

• The Most Accurate Apple iPhone Review To Date From Maddox
• Google Helps Break Down The Language Barrier (In A Brilliant Way)
• Sun buys MySQL for $1bln!
• Getting The Most Out Of The MySQL Conference
• Sun Definitely Developing A Phone This Year

• the database
• CPU
• memory
• IO
• disk latency
• network latency
• slow queries

• 300 platforms (300 remote agents collecting data)
• 2,100 servers
• 21,000 services (10 services per server), sounds feasible
• 468,000 metrics (20 metrics per service)
• 28,800,000 metric data rows per day
• larger deployments have a lot more of these (sounds crazy)

• measurement_id
• timestamp
• value
• primary key (timestamp, measurement_id)

• agent collects data and sends reports to server with multiple data points
• server batch inserts metric data points
• if network connection fails, agent continues to collect but server "backfills" unavailable
• when agent reconnects, spooled data overwrite backfilled data points (why not use REPLACE for all inserts?)

• INSERT INTO TABLE (a,b,c) VALUES (0,0,0), (1,1,1),…
• using MySQL batch insert statements vs prepared statements with multiple queries in other databases seems to improve overall performance by 30%
• batch inserts are limited by 'max_allowed_packet'

• set unique_checks=0, insert, set unique_checks=1 (definitely need to make sure data is valid first)
• set foreign_key_checks=0, insert, set foreign_key_checks=1 (same concerns as above)
• Hyperic doesn't use the 2 above

• when regular INSERT fails, retry batch with INSERT ON DUPLICATE KEY syntax
• it's much slower but it allows

• queue metric data from separate agent reports
• minimize number of inserts, connections, CPU load
• maximize workload efficiency
• optimal configuration for 700 agents
• 3 workers
• 2000 batch size seems to work best
• queue size of 4,000,000
• this seems to peak at 2.2mil metric data inserts per minute

• inspired by rrdtool
• lower resolution tables track min, avg, and max
• data compression runs hourly
• size limit 2 days
• every hour, data is rolled up into another table that holds hourly aggregated values with size limit 14 days, then that one gets rolled up into a monthly table, etc
• this is is a good approach if you don't care about each data point

I'm overwhelmed by the amount of "you know"s from the speaker. Parasite words, ahh! Sorry Charles

• measurement data split into 18 tables, representing 9 days (2 per day)
• they didn't want to do more than 2 SELECTs to get data per day, hence such sharding
• oddly, Charles didn't actually use the word 'shard' once
• tables truncated, rather than deleting rows => huge performance boost
• truncation vs deletion
• deletion causes contention on rows
• truncation doesn't produce fragmentation
• truncation just drops and recreates the table - single DDL operation

• every InnoDB table has a special index called the clustered index (based on primary key) where the physical data for the rows is stored
• advantages
• selects faster - row data is on the same page where the index search leads
• inserts in (timestamp) order - avoid page splits and fragmentation
• shows comparison between non-clustered index and clustered index (see slides)

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Conference Liveblogging: Applied Partitioning And Scaling your (OLTP) Database System (Wednesday 11:55AM)
• MySQL Conference Liveblogging: Introduction To The BLOB Streaming Project (Wednesday 3:00PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

• an enum is both a string and a number
• sorted by number
• displayed as string
• can be sorted by string if it's cast as string

• compacts storage
• faster to search
• if (var)char is turned into enum, some space can be saved, shows example

• order by rand(1) - obviously
• the integer parameter is actually a seed
• really slow, also obviously, especially for larger tables because it has to order first, then apply rand() to the list
• another solution is to add an extra column, put random values into it, and add index, then
• UPDATE tbl SET chaos=RAND();
• random extraction
• SELECT id, col FROM tbl JOIN
    (SELECT CEIL(RAND() *
      (SELECT MAX(id) FROM tbl)) AS r)
  AS r2 ON id=r;
• for multiple rows, Damien offers a solution involving an integer table (see his own slides). This solution is supposed to be really fast.

• generate a table with only 10 integers
• do cross joins to get 100, 10,000, etc integers
• 10mil rows takes only 13 seconds (on his laptop), then either store it in another table or just use the result set
• yet again, code is shows, so see slides

• shows a handy query

• offers a long query that uses ELT(), CONCAT(), and RAND() (see slides)
• a SELECT that builds words uses GROUP_CONCAT()
• looks pretty useful

• SELECT version, REPEAT('*', percentage * 2) AS bars FROM php_versions;
• shows a nifty result set with stars as fillers in percentage bars
• shows another result set, similar to the one above, with spaces as fillers and stars at the end
• shows another example that looks like some sort of a spaceship, from mysql forge

• CONCAT() and CONCAT_WS() now for groups
• ORDER BY
• SEPARATOR
• limited to 1kb by default, change group_concat_max_len
• shows example (see slides)

• uses GROUP_CONCAT, looks horribly long and inefficient, there has to be a better way
• somebody suggests using ORDER BY with LIMIT

• uses MySQL vars
• not as many people as I thought seem to know about it, judging by the low amount of raised hands
• SET @var:=3
• SELECT @var;
• variables are not shared between connections, destroyed at the end of the connection
• uses LEAST()
• see slides

• SET @i:=0
• LOAD DATA INFILE '/tmp/stats.txt'
  INTO TABLE statPHPload
    (@date, @php, @country, @continent)
  SET
    id = 0,
    period = date(STR_TO_DATE(@date, '%d-%b-%y')),
    rank = (@i := @i + 1),
    php = CAST(REPLACE(@php, ',', '.') AS DECIMAL),
    country = @country;

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: Applied Partitioning And Scaling your (OLTP) Database System (Wednesday 11:55AM)
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)
• MySQL Conference Liveblogging: MySQL Hidden Treasures (Thursday 11:55PM)

• MySQL Conference 2008
• Navicat For MySQL Bugs Filed
• MySQL Conference Liveblogging: Benchmarking Tools (Wednesday 4:25PM)
• MySQL Conference Liveblogging: Monitoring Tools (Wednesday 5:15PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)