curl -IL "URL"

This command would send a HEAD request (-I), follow through all redirects (-L), and display some useful information in the end. Most of the time it's ideal:

curl -IL "http://www.google.com"
 
HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 03:58:55 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Server: gws
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked

However, the server I was curling didn't support HEAD requests explicitly. Additionally, I was really only interested in HTTP status codes and not in the rest of the output. This means I would have to change my strategy and issue GET requests, ignoring HTML output completely.

Curl manual to the rescue. A few minutes later, I came up with the following, which served my needs perfectly:

curl -sL -w "%{http_code} %{url_effective}\\n" "URL" -o /dev/null

Here is a sample of what comes out:

curl -sL -w "%{http_code} %{url_effective}\\n" "http://www.amazon.com/Kindle-Wireless-Reading-Display-Generation/dp/B0015T963C?tag=androidpolice-20" -o /dev/null
 
200 http://www.amazon.com/Kindle-Wireless-Reading-Display-Generation/dp/B0015T963C

Here, -s silences curl's progress output, -L follows all redirects as before, -w prints the report using a custom format, and -o redirects curl's HTML output to /dev/null.

Here are the other special variables available in case you want to customize the output some more:

• url_effective
• http_code
• http_connect
• time_total
• time_namelookup
• time_connect
• time_pretransfer
• time_redirect
• time_starttransfer
• size_download
• size_upload
• size_header
• size_request
• speed_download
• speed_upload
• content_type
• num_connects
• num_redirects
• ftp_entry_path

Is there a better way to do this with curl? Perhaps, but this way offers the most flexibility, as I am in control of all the formatting.

• How To Show Hidden/Invisible Files In Total Commander, Both Locally And On An FTP Server
• How *Not* To Implement A Web Application That Handles External Authentication, Using BeTwittered.com As An Example
• Swapping Column Values in MySQL
• Wireless Headphones Are Nothing New – What About Wireless Earbuds?
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)

The Problem

The problem with ExpandDrive is – it's quite simplistic. So simplistic that it doesn't offer an easy way to export its drive list and the associated settings. Because of that, you will have to enter all the drives all over again in case you reinstall Windows or want to replicate them to another computer.

Of course, there is a workaround.

The Workaround

The workaround involves the following easy steps and, as I don't have a Mac, works only on Windows:

1. On the source computer, load the registry editor by opening the run menu (WinKey+R) and typing in 'regedit'.

2. Expand the following location: HKEY_CURRENT_USER\Software\ExpanDrive. If you have SFTPDrive, install ExpanDrive first, which should migrate the old settings to this location.
Inside, you should find these 2 subfolders: Hostkeys and Sessions

3. Right click on ExpanDrive, select Export, and then save the reg file as expandrive.reg while making sure the "Selected branch" is selected in the Save dialog:

4. Close ExpanDrive on your new computer, transfer expandrive.reg to it, and then execute it. Click "Yes" at the following dialog:

5. Reopen ExpanDrive and you should now have all your drives.

If anyone has a better method, comments about the one above, or a way to carry over the public/private key settings, please share in the comments.

• How To Resize/Grow VMware Linux Disks and Partitions
• Is Your Simplifymedia For Winamp Broken On A 64 Bit Windows 7? Here's How To Fix It
• StackOverflow.com, SuperUser.com, ServerFault.com Fan? Here's A Greasemonkey Script To Keep Track Of All Your Accounts
• [Android] Auto Formatting Android XML Files With Eclipse
• How To Fix Incomplete WordPress (WXR) Exports

I was really glad to see Joel Spolsky, Jeff Atwood, and the whole StackOverflow team in person, as well as listen to great talks in the following topics:

9:00 – 9:50    Joel Spolsky Opening Keynote
9:50 – 10:45    Mark Harrison Python
11:00 – 11:55    Rory Blyth iPhone
11:55 – 12:25    Joel Spolsky Fogbugz
13:30 – 14:25    Scott Hanselman ASP.NET-MVC
14:25 – 14:45    Jeff Atwood Stack Overflow
14:45 – 15:40    Daniel Rocha Qt
16:10 – 17:05    James Yum Android
17:05 – 18:00    Yehuda Katz jQuery

My own favorite topics were in the following order of fun/usefulness level:

• iPhone (though I'm interested in Android development myself but it's always good to check out the competition) – it was fun and not boring. Quirky, jittery, OCD all come to mind.
• Joel's keynote (Hi, Joel, I'm @ArtemR ) – I welcome the discussion of simple vs feature rich, though Joel has talked about this on his blog multiple times.
• jQuery – nice intro into history of jQuery and competition with other frameworks. The technical level of specific examples was quite basic.
• Android – this is the topic I'm mostly interested in and unfortunately the main speaker got substituted by a guy who has obviously not presented in front of a large crowd before (sorry, I have to criticize – you'll do better and better each time! Just don't worry so much and know your stuff) and was pretty much a recent college grad from the looks of it. He did a pretty decent job at making the code run but a really poor job of marketing and explaining Android's weaknesses and strengths, as some people on twitter pointed out.
• StackOverflow review – quite brief but fun. Thanks Jeff!
• Python – very odd that it was even there – just a look at a few lines of code from the infamous Python google-like suggest algorithm.
• Qt – didn't really care for it but sat through it. Nokia, meh. Next!
• ASP.NET, Fogbugz – I skipped as I didn't care much for those

Overall, I feel like the information presented in all sessions was pretty basic but I guess that was the idea – a little bit of everything for everyone.

The absence of WiFi was pretty disappointing as I think it was advertised but I didn't care much for it. Kept everyone more concentrated anyway.

The free memory upgrades by Microsoft was an AWESOME move (though mine is already at 4GB and they didn't give out 4GB sticks)!

Lunch was decent – Boudin. Lots of drinks and snacks.

Well worth $99.

Twitter #devdays results.

Cross-posted to Meta-StackOverflow.

• Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]
• StackOverflow.com, SuperUser.com, ServerFault.com Fan? Here's A Greasemonkey Script To Keep Track Of All Your Accounts
• Monitor All Your Domains From One Location
• The Real Reasons To Use Twitter (Get Over Your Prejudice Already)
• Minority Report Becomes Reality

Screen is awesome. Once you become comfortable navigating around it, you start using it ALL the time. No more dropped sessions, no having 10 Putty windows open at the same time, no more nohup.

However, with default screen settings I've always felt a bit lost and out of place, mostly because there was no "taskbar" with a bird's eye view of all windows. Pressing ctrl-a, " really does get annoying fast (that's the command that brings up the window selector – screenshot below).

So instead I modded my screen to have a "taskbar" which sits at the bottom of screen and adds:

• the name of each window
• a clear marking of window states, such as
  • the currently active window (* and different background)
  • last used window (-)
  • if you're sharing the current window with someone else (&)
• the current incoming/outgoing bandwidth in bytes/sec, updating every 5 seconds
• the current time
• the load on the server, as reported by the uptime or w commands

Of course, all of the above is customizable to your liking, but this is what I picked for myself.

Without further ado, here's a screenshot:

Doesn't that make you feel much more homey?

Now using a shared screen session for pair programming, as suggested in a recent article posted on Hacker News, suddenly sounds even more appealing.

Installation

Here's what you need to get this bar on your screen:

• download my .screenrc (0 B, downloaded 963 times) and put it in your home directory. Make sure the location of wormulon (installed in the next step) is correct. Also, specify the right network interface (eth0 in most cases).
  Here is the file, for reference:

1
2
3
4
5
6
7
8
9
10
11

# Monitor eth0 bandwidth and update every 5 s
# Assign the output of this cmd to '%1`' and use it in the below hardstatus line:
backtick 1 0 0 /usr/local/bin/wormulon -d 5 -i eth0
 
hardstatus alwayslastline "%{bw}[%H] [%?%-Lw%?%{wb}%n*%f %t%{bw}%?%+Lw%?]%=%{bw} [%1`] [%c:%s] [%l]" # modified from http://lists.gnu.org/archive/html/screen-users/2007-08/msg00008.html
 
# Set the scrollback length:
defscrollback 10000
 
# Select whether you want to see the copyright notice during startup:
startup_message off

• download and compile wormulon (wormulon-0.1.4.tar.gz 45.3 kB, downloaded 473 times or from a mirror) – a nifty little program that shows incoming and outgoing bandwidth and is perfect for integrating with screen. You can omit this step if you don't want the bandwidth monitor. I know the name is a bit odd but it's a legit little open source utility – feel free to check out the code.

tar xvzf wormulon-0.1.4.tar.gz
cd wormulon-0.1.4/
./configure && make
sudo make install # this installs into /usr/local/bin/;
                  # alternatively, copy the wormulon binary wherever you
                  # want and update the path in .screenrc

Now restart screen and you're good to go.

• How To Download And Install Windows Media Player 11 Bypassing WGA
• Watch – A Useful Linux Command You May Have Never Heard Of
• How To Update Eclipse From Galileo (3.5) To Helios (3.6) In-Place Without Reinstalling
• Mastering The Linux Shell – Bash Shortcuts Explained (Now With Cheat Sheets)
• How To Resize/Grow VMware Linux Disks and Partitions

I am throwing up a quick post about a relatively cryptic error that Solr started throwing the other day here at Plaxo. After happily running for a few days, I suddenly started getting pages about failed Solr indexing.

Upon closer examination, I saw the following repeatedly in the log file:

catalina.2009-09-18.log:SEVERE: java.io.IOException: directory 'DATADIR/index'
exists and is a directory, but cannot be listed: list() returned null

I tried to see if sending an OPTIMIZE command would help but the server returned the same response.

Digging Deeper

The reason was these errors was quite simple – Solr was running into the system level limit on allowed number of open files (ulimit). This limit can be seen by running

ulimit -n
1024

or simply

ulimit -a | grep 'open files'
open files                      (-n) 1024

This means that if a process tries to open that many files at the same time, the kernel will prohibit opening any more, which in my case caused the Java IOException.

In my case, I haven't been using the Solr OPTIMIZE command for a while, so after a lot of COMMITs, the Solr data got pretty fragmented, thus hitting the open files limit.

The Solution

There are 2 things to be done here:

1. OPTIMIZE Solr more often. When an OPTIMIZE occurs, multiple index files are merged into 1, thus reducing the number of files that need to be opened. However, before you can OPTIMIZE, you have to raise the allowed number of open files (see the next bullet).

2. Set a higher open files limit for the user that runs Solr (in my case, the solr user) – for example to 4096 instead of 1024. One way to do it is by adding a file /etc/security/limits.d/solr.conf with the following contents:

   solr hard nofile 4096
   solr soft nofile 4096

   and then logging out and back in. The file should be automatically loaded, which you can verify by running the ulimit commands from the section above.

Happy Solring!

By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.

• How To Show Hidden/Invisible Files In Total Commander, Both Locally And On An FTP Server
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• Watch – A Useful Linux Command You May Have Never Heard Of
• [Perl] Finding Files, The Fun And Elegant Way
• How To Sort Folders The Same Way As Files In Total Commander

Every time I have to deal with svn:externals in SVN, I forget the command line syntax. Every single damn time. Normally, I use SVN GUI clients, such as SmartSVN, which make it very simple to add an svn:externals property. But for command line, it always takes looking at 25 different sites on google, which are all incredibly unhelpful for this question for some reason. Trying "svn help propset" on the command line was bloated and equally useless.

So this time I needed to write it down and make sure everyone who needed help with svn:externals would find exactly what they need here. I hope this page will soon come up on top of all the unhelpful results on google for "propset svn:externals" and other related queries.

The Problem

I want to set a simple svn:externals property in one of my project's directories, lets say 'plugins' (talking about WordPress here). The outcome would be a directory called 'akismet' within 'plugins' that points to a remote svn url.

Various combinations of trying to do it produced pathetic results, like

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

svn propset svn:externals . akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Error parsing svn:externals property on 'akismet': '.'

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk akismet
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

The Solution

Finally, thanks to this post, I found the right command:

svn propset svn:externals 'akismet http://plugins.svn.wordpress.org/akismet/trunk' .
property 'svn:externals' set on '.'

Note that dot at the end of the command and the quotes around the directory name and url.

Now commit via

svn commit

and then

svn up
Fetching external item into 'akismet'
A    akismet/akismet.gif
A    akismet/akismet.php
A    akismet/readme.txt
Updated external to revision 127962.
 
Updated to revision 16.

There, was it that hard, forum gurus and blog fiends?

Edit: in order to set multiple directory/url pairs in a single svn:externals property, you should put the individual dir/url pairs into a file (let's call it 'svn.externals'), like so

akismet http://svn.wp-plugins.org/akismet/trunk
all-in-one-seo-pack http://svn.wp-plugins.org/all-in-one-seo-pack/trunk

and then apply the property using

svn propset svn:externals -F svn.externals .

You should also just check in 'svn.externals' to easily keep track of it.

Introduction

Whether you are a web developer or a self-hosting business owner, the only excuse for not activating compression capabilities of your web server can be that you didn't know about it. And now that you are reading this, there is no excuse left at all.

Here is how big a single page of this blog was before compression was enabled on CSS and Javascript files (computed by YSlow):

And here it is after compression:

As you see, the difference is quite substantial – almost 30% savings.

Compressing your HTML, XML, Javascript, CSS, etc pages will mean less data transferred between the server and the client which:

• reduces the bandwidth usage.
• provides faster page rendering which in turn leads to less user frustration, higher conversion rates, lower bounce rate, etc etc etc.

Compression is especially important for users with slow connections as every kilobyte of your code is that much more painful to them.

Compression can be very effective – you can easily shrink your text, code (HTML, XML, Javascript, CSS, etc) to 10% of the original size (of course, your mileage may vary). 100KB page that needs only 10KB to transfer? Sign me up!

So, before I talk about the solution, let me describe what exactly happens when compression is turned on and how it affects older browsers that don't support it.

Compression Mechanism Explained

In order for compression to work in the first place, the web server (Apache in my example) needs to support it. This is achieved by enabling one of Apache modules called mod_deflate. The server will then be able to compress the data to the DEFLATE standard using either the zlib (also known as deflate) or gzip implementations. Yeah, DEFLATE is both the standard the one of its implementations, for those confused. I know I was. This is best described in this Wikipedia article.

The following mechanism is used:

• the server with a compression extension enabled is able to serve either compressed (smaller) or uncompressed (larger) pages, depending on what the client supports.
• the client (that is, your browser) sends a special header called "Accept-Encoding" listing the DEFLATE implementations it's capable of decompressing. For example "gzip,deflate".
• the server picks the best compression supported by the client (if any), compresses the files, and sends them over to the client.
• the client receives the compressed files and decompresses them.

Are Your Pages Already Compressed? Test Them!

If you are not sure whether you are already serving compressed pages or not, test them! My favorite way is by using Charles HTTP Debugger. Another option is by downloading Firebug for Firefox and installing Yahoo's YSlow or Google's Page Speed. Just look at the response headers to see if compression is on (look for the Content-Encoding header). Here are some before and after examples:

Theme CSS

Before:

After:

jQuery

Before:

After:

Solution

Create a .htaccess file in the top directory of your site with the following contents:

1
2
3
4
5
6
7
8

# DEFLATE by type - html, text, css, xml
AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml
 
# DEFLATE by type - javascript
AddOutputFilterByType DEFLATE application/x-javascript application/javascript text/javascript text/x-js text/x-javascript
 
# DEFLATE by extension
AddOutputFilter DEFLATE js css htm html xml

Alternatively, you could put these lines into your Apache config within the Directory directive.

The AddOutputFilterByType directive adds DEFLATE filters to certain MIME types. I tried to assemble some of the common ones but feel free to add more, as each server may be configured differently and give out MIME types different from mine.

The AddOutputFilter directive binds the DEFLATE filter to specific file extensions, just in case they are not served with a proper MIME type. Feel free to add to this list as well.

Caveats

1. In order to use this whole compression/deflate/gzip business, your Apache server must first have mod_deflate enabled. Without it, you will get the HTTP 500 error (Internal Server error). You can check which mods you already have enabled by checking with the output of phpinfo() function.

In order to enable mod_deflate, uncomment the line with "deflate_module" in your Apache config file. The location of this config file is highly dependant on your system. Some examples include

• /etc/apache2/httpd.conf
• /etc/httpd/conf/httpd.conf
• c:\wamp\bin\apache\Apache2.2.11\conf\httpd.conf
• some other place where your system stores Apache config files (read the special note below for OpenSUSE).

Here's what you should have:

LoadModule deflate_module modules/mod_deflate.so

On OpenSUSE, you actually enable modules a bit differently. Go to /etc/sysconfig/apache2 and look for APACHE_MODULES=. Then add "deflate" to the list, if it's not already there.

Now, restart Apache and check the output of phpinfo() again.

2. Adding AddOutputFilter and AddOutputFilterByType to .htaccess requires such overrides to be authorized by the main Apache configuration for that directory, otherwise it will return error 500 as well. The option you are looking for is called "AllowOverride" and mine was set to "AllowOverride AuthConfig" which wasn't enough. Changing it to

AllowOverride AuthConfig FileInfo

or just

AllowOverride All

fixes the problem. You can find more info about AllowOverride here.

3. In WordPress, if you are using Google Gears (Turbo mode) for caching some core WordPress files, they will not show up compressed. That is because they're not served by the remote server but rather reside locally (think of it as permanent cache). I was very confused at first when I didn't see jQuery.js in the HTTP log and YSlow reported it uncompressed.

Are you a WordPress user?

If you are a WordPress user, don't assume WordPress is going to automatically compress your pages. In fact, as you install more and more plugins, the payload becomes larger and larger with those additional CSS and Javascript files.

You owe it to yourself and to your users to immediately enable compression on your blog.

Here is what happened after I enabled compression on this blog.

Before:

After:

Bonus – WP Minify

For even better results, I suggest you have a look at my good friend and talented WordPress master Thaya's plugin called WP Minify. It preprocesses and aggregates all or most of your CSS and Javascript into just 2 files, thus saving on the number of HTTP requests. It also minifies content to achieve smaller size.

My blog before WP Minify:

After WP Minify:

That's all folks. Let me know if something was unclear and I'll be glad to clarify it.

A few references that pointed me in the right direction and allowed me to provide a more complete solution:

• http://brightscape.net/compress-jquery-even-further/
• http://brightscape.net/compress-your-web-pages-with-mod_deflate/

• Beer Planet Upgraded To WordPress 2.5
• [WordPress Plugin Development] How To Include CSS and JavaScript Conditionally And Only When Needed By The Posts
• Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]
• Rendr – a Live Awesome CSS/HTML Rendering Tool
• cpan – The Perl Module Manager

Let me briefly discuss some of the other methods first.

Limited

Using glob() (or <>, TODO verify) you can find files in a single directory, using only the limited shell wildcard support. For example,

1

my @files = glob("tmp*");

Boring

File::Find is the de facto standard for searching in Perl.

This method finds files that end in .pl in "." and "../SomeDir", following symlinks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/perl -w
 
use File::Find;
use Data::Dumper;
use File::Basename;
my @directories_to_search = (".", "../SomeDir");
my @file_list = ();
 
find(
  { wanted =>
    sub {
      if ( basename($File::Find::name) =~ /\.pl$/i )
      {
        push @file_list, $File::Find::name;
      }
    },
    follow => 1
  },
  @directories_to_search
);
print Dumper @file_list;

It works fine, except it's horribly ugly and boring. Let's have a look at something more fun.

The Fun And Elegant Way

File::Find::Rule. Just have a look at this beauty.

Just like above, find all .pl files in "." and "../SomeDir", following symlinks:

1
2

print Dumper (File::Find::Rule->name("*.pl")->file->extras({ follow => 1 })->
in(".", "../SomeDir"));

Same as above, except bypass .svn directories (shaves off a ton of time with a lot of directories):

1
2

print Dumper (File::Find::Rule->not(File::Find::Rule->directory->name('.svn')->
prune->discard)->name("*.pl")->file->extras({ follow => 1 })->in(".", "../SomeDir"));

Find all .log files that are older than 24 hours in "."

1
2
3

my $epoch_time_1_day_ago = time() - 60*60*24;
print Dumper (File::Find::Rule->file->name("*.log")->
mtime("<$epoch_time_1_day_ago")->in('.'));

Be sure to read the File::Find::Rule perldoc for more options and remember: have fun with your code!

Thanks to Perlbuzz and Andy Lester for pointing me to this library a few months ago.

• [Perl] How To Get The Path Of An Included Library (.pm), Regardless Of Current Directory
• FTPRush Cleanup Script
• Quick Perl Snippet: Finding If A File Has A Media Extension Using Regex
• How To Check If The Local SVN Revision Is Up-To-Date
• How To Sort Folders The Same Way As Files In Total Commander

If you, like me, are building or thinking of implementing a MySQL-powered application that has any need for prioritizing selecting certain data over other data, this article is for you.

Example

As a real world example, consider a queue-like video processing system. Your application receives new videos and processes them. The volume of incoming videos can at times be higher than the processing rate because the process is CPU bound, so occasionally a pretty long queue may form. You will try to process them as fast as you can but…

…suddenly, you need to process a video somewhere in the middle of the queue or an important video enters and needs immediate attention. What do you do?

An obvious solution is implementing a simple priority system where each item has a numeric priority field. Now you can sort first by priority from highest to lowest and then by id within the highest priority. Important and urgent items get a their priority changed to something higher and get processed first. There is only one problem.

Problem

The problem is pretty serious – let’s take a look at the SELECT statement. Before selecting, I’ve added 19 random rows to have some data to work on.

SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;

What kind of index would you put on this table to speed up this query? You do want to add a proper index, don’t you? DO YOU? Ok, good.

Here’s what happens without any indexes:

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
|  1 | SIMPLE      | queue | ALL  | NULL          | NULL | NULL    | NULL |   19 | Using filesort |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)

Using filesort, ugh, of course, due to sorting without an index.

Let’s see, how about a combined index on (priority, id)?

mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority`, `id`);
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

Better because an index is being used but not very good because filesort is still present. “Of course!”, you slap yourself on the forehead. The first ORDER BY uses a DESCENDING order, and our key is in ASCENDING order.

So, let’s add the proper key with the right ordering instead.

mysql> ALTER TABLE `queue` DROP INDEX `priority_id`;
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority` DESC, `id`);
Query OK, 19 rows affected (0.06 sec)
Records: 19  Duplicates: 0  Warnings: 0

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

What the deuce? This is the same result as with the previous index. Time to dig up the documentation.

Here is what the MySQL manual has to say under the ORDER BY optimization section:

MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause … if you mix ASC and DESC:

SELECT * FROM t1 ORDER BY key_part1 DESC, key_part2 ASC;

Moreover, to confuse the user even more, the index creation command accepts the DESC instruction, without actually honoring it, as specified in the CREATE INDEX section:

An index_col_name specification can end with ASC or DESC. These keywords are allowed for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

So, after so many years MySQL still doesn’t support such basic functionality – you are either stuck with a query that uses filesort or have to look for a workaround.

Solution

Since it’s not possible to mix order directions, the solution is then to change the meaning of the priority column to match your needs. Thus, in the new approach priority 1 is higher than priority 10, and the application logic needs to accommodate to that. If you caught this while the application is still young, the code may be easy to change, but otherwise it could be a major pain in the butt.

Conclusion

The moral here is: plan your queries ahead and don’t mix and match DESC and ASC ordering as MySQL will not be able to use an index to resolve it. Do it even sooner if you’re putting lots and lots of data into your tables.

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

While writing a 1093985th Perl script the other day I was facing the following dilemma:

• Let’s say there is a local library, called TheUberLib.pm. It is so uber that most of my scripts, located all over the machine, include it.
• Now, let’s also say that there’s an even more uberly important binary called run_me_now_or_you_will_die but the only way to find it is by using a relative path to the aforementioned TheUberLib.pm, for example ../bin (RELATIVE TO TheUberLib.pm).
• I don’t want to hardcode the path to run_me_now_or_you_will_die because it can be different on multiple machines and the code wouldn’t be robust enough – all I know is that the path is relative to an included library.

So how does one get the path to an included TheUberLib.pm (in my case to deduce the path to run_me_now_or_you_will_die), regardless of the script location and the current directory?

I started looking at all possible ways of getting this information.

• At first I looked at $0. $0 returns the path to the Perl script itself (LibraryPath/lib_path.pl) and is completely useless here.
• The FindBin module that I sometimes use was also of no help as it also deals with the caller script path.

Solution

After digging around for a bit, this is the method I came up with that did exactly what I wanted:

1
2

use File::Basename;
my $path_to_uber_lib = dirname($INC{'TheUberLib.pm'});

and the clean way to get the final path to run_me_now_or_you_will_die is:

1
2
3

my $relative_path_to_bin_dir = "../the_path_we_want/run_me_now_or_you_will_die";
my $path_to_deadly_bin = Cwd::realpath("$path_to_uber_lib/$relative_path_to_bin_dir");
print "The path to deadly bin is: $path_to_deadly_bin\n";

which prints /tmp/LibraryPath/some_path1/the_path_we_want/run_me_now_or_you_will_die.

Here is the full program with all directories that you can check out from SVN:

svn co http://beerpla.net/svn/public/Perl/LibraryPath/

or just browse the code here: http://beerpla.net/svn/public/Perl/LibraryPath/

Explanation

The %INC hash contains all included libraries, with library names as keys and full paths as values.

The dirname() function is part of File::Basename and returns just the directory part of the path.

Finally, Cwd::realpath() is a function that resolves relative and symbolic links to canonical absolute ones.

And there you have it.

• FTPRush Cleanup Script
• How Do I Get Both The Return Value And Text In Perl? Backticks vs. System() (Perl 5.10)
• [Perl] Finding Files, The Fun And Elegant Way
• Parsing JSON In Perl By Example – SouthParkStudios.com South Park Episodes
• [WordPress Plugin Development] How To Include CSS and JavaScript Conditionally And Only When Needed By The Posts

Here are my findings:

1. The

   UPDATE swap_test SET x=y, y=x;

   approach doesn't work, as it'll just set both values to y.

   PostgreSQL seems to handle this query differently, as it apparently uses the old values throughout the whole query. [Reference]

2. Here's a method that uses a temporary variable. Thanks to Antony from the comments for the "IS NOT NULL" tweak. Without it, the query works unpredictably. See the table schema at the end of the post. This method doesn't swap the values if one of them is NULL. Use method #3 that doesn't have this limitation.

   UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL;

   The parentheses around @temp:=x are critical. Omitting them will cause data corruption.

   1 2 3

   mysql> UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

3. This method was offered by Dipin in the comments. I think it’s the most elegant and clean solution. It works with both NULL and non-NULL values.

   UPDATE swap_test SET x=(@temp:=x), x = y, y = @temp;

4. Another approach I came up with that seems to work:

   UPDATE swaptest s1, swaptest s2 SET s1.x=s1.y, s1.y=s2.x WHERE s1.id=s2.id;

   1 2 3

   mysql> update swap_test s1, swap_test s2 set s1.x=s1.y, s1.y=s2.x where s1.id=s2.id; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

Essentially, the 1st table is the one getting updated and the 2nd one is used to pull the old data from.

Test schema used:

1
2
3
4
5
6
7
8
9
10

CREATE TABLE `swap_test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `x` varchar(255) DEFAULT NULL,
  `y` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;
 
INSERT INTO `swap_test` VALUES ('1', 'a', '10');
INSERT INTO `swap_test` VALUES ('2', NULL, '20');
INSERT INTO `swap_test` VALUES ('3', 'c', NULL);

Do you have a better approach? If so, please share in the comments.

Some references:

• http://stackoverflow.com/questions/37649/swapping-column-values-in-mysql/ – some discussion on various methods, which eventually prompted me to start this post
• http://www.marcworrell.com/article-3026-en.html – discussion on the 2nd approach, which doesn’t work

• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• How To Diagnose And Fix Incorrect Post Comment Counts In WordPress
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations

Everyone and their mother are throwing out their predictions for 2009 nowadays, it’s a new fad. It’s like you’re not cool anymore if you don’t have twitter, a Mac, and a set of random predictions for the next 12 joyous months.

So I decided to throw in a few ideas of my own to be part of the cool crowd again (how much cooler can I be already, you might think, and I wouldn’t blame you).

Disclaimer (read it, tough guy)

What this post is:

• about the future of technology and the Internet, 2009 and beyond.
• my ideas on what is going to happen or should happen. If they happen to match someone else’s ideas – it doesn’t mean I ripped them off, it just means we share the same opinions and they’re more likely to come true.
• awesome.

What this post is not:

• predictions I pulled out of my ass, like “the market will bounce in August 2009” because some random douche said so.
• a collection of stolen ideas. I have reserved a separate post for that purpose.
• a raptor on hoverboard.

Things That Need To Happen

1. Socially Editable Maps

With the advent of the GPS technology in the last couple of years and GPS prices falling (my originally $800 Garmin Nuvi 660 now costs about $200), the biggest frustration I have now is the accuracy of information. In the world of Google Maps and Wikipedia, why is it that I have to wait a whole year for map updates that are obsolete by the time they come out? The Bay Bridge repairs change the roads on an almost monthly basis, for example.

I want to know about road changes as soon as they occur.

The next big thing will be a company, either existing, like Google, or a startup, that will introduce the social aspect into the mapping technology. It will do for maps and GPS what Wikipedia did for text, using the same approach. The details are of course to be worked out.

After this concept becomes successful, GPS companies, will need to support such updates over the air, with a push of a button. This ties in closely with prediction #2 and #4.

2. Open Source GPS Goes Mainstream

Google, the father of Android, needs to get behind this one as well. In fact, an open source GPS device would really be a subset of Android’s functionality, in a dedicated device, so it shouldn’t be that hard. In the far future, it will be built into automobile dashboards but I don’t foresee that happening in the near future.

Combined with prediction #1, this will be a killer device. I really believe that free, open source software is the wave of the future, the natural direction of where software development is heading. Look at Linux in the past few years. Look at Android now. Release one and I’m be #1 in line for one. Those who know me IRL know that I can’t go anywhere without a GPS anymore. I once forgot a GPS at home and ended up in Chili.

3. Photo and Video Cameras Will Go Wireless, Anywhere You Go

Photo and video cameras will be able to post pictures and videos to your home computer, sites like Flickr and YouTube, or email themselves directly to your friends… as soon as you record them, over cellular and Wifi networks.

Cell phones with cameras? Think cameras with cell phones. Some Wifi enabled cameras already came out, like this Canon SD430 (DP review here), and this freshly announced at CES 2009 Sony G3, and now it’s time to go cellular.

The next prediction is a generalization of this concept.

4. Wireless Connectivity Everywhere, In Every Device, Anywhere You Go

Cell phone carriers will sign an increasing amount of deals with companies that want to build devices which connect to the Internet or private networks anywhere you go: GPSes, cameras, digital picture frames, cars, fridges.

The future is completely wireless. People are sick and tired of cords, clutter, and Rick Astley – I know that for a fact and digg says so.

5. Android Will Become Hugely Successful

Alright, I’m dead tired of all the posts about how Google’s Android is going to flop in 2009. I sincerely feel that people with that point of view have never experienced Google and open source, played with Android itself, or realized what exactly an open source phone OS means to developers, manufacturers, and consumers.

Android will become a platform of choice in the next couple of years as everyone starts to realize its limitless possibilities, the OS matures, thousands of new apps get written for it, and new Android phones flood the market. The only Android phone released so far, HTC’s G1, has already received praise from consumers, as well as my friends (even the females ones are excited. And yes, I have female friends).

Motorola announced a few months ago that its new phones are going to run Android and Windows Mobile exclusively. If a company with an R&D department as big as Moto’s stands behind something, you bet your ass they’ve done they’re homework. And what’s not to like? They can now:

• take something that’s supported by the biggest Internet company in the world and the community, for free.
• stop worrying about writing and maintaining their own OS – BAM, tons of money saved.
• concentrate R&D on the hardware.
• if need be, freely develop the features they want that Android doesn’t support yet and contribute them back. Everyone wins, except maybe Symbian developers that don’t have a job anymore.

Android is going to be a revolution. Apple and Android fighting it out will be the best thing that happened to us since the invention of sushi.

6. Open Source HDTVs

The open source revolution will continue and start penetrating the HDTV market.

Personally, I think set top boxes are a waste of effort, time, and money but give me an HDTV that can run Youtube, Vimeo, LastFM, Pandora, and any other site through some sort of a plugin or browser, with a build in Media Center that connects to my computers and goes Wifi, that uses open source, upgradeable software (most likely Linux based), and I will buy it in a heartbeat.

Yes, what I’m saying is if my TV and HTPC did something dirty, I would totally dig their offspring, and so would millions of other people who don’t want/need/care/understand HTPC. TV in general holds a special place in my heart, and make it an even better experience, people.

7. Twitter’s Popularity Will Explode

Now for some social media. Twitter will continue dominating the microblogging arena, similar to YouTube dominating the video space. The site will grow extremely fast, much faster than now.

Competitors, like plurk, will probably secure certain small niches but nobody will be close to touch twitter.

Twitter’s significance in the business world for will be revolutionary. It will become second nature for every company to have one or many twitter accounts as means of connecting to consumers on a personal level. Think an opportunity for mini press releases, many of them, daily, not boring ones, the ones people are actually going to read.

8. Social Media Jobs Will Be In Increasing Demand

SEO, social marketing, and viral campaigns are very cost effective ways of brand promotion, and every company will want to jump on board.

As more and more of them realize this, they will start needing more people with marketing skills of John Chow, Chris Brogan, Nicky Cakes (this dude is hilarious), Jeremy Schoemaker, and the like.

9. YouTube Will Continue To Dominate

This one is easy:

• Biggest user base.
• HD. My favorite greasemonkey plugin YousableTubeFix exposes hi/lo FLVs, MP4, and HD MP4 options. Better quality equals better user experience.
• Chromeless player. YouTube is the only company I know of that has a Javascript controlled chromeless player, which can be embedded in any other flash player. Combined with already existing millions of embeds all over the Internet, YouTube’s popularity isn’t going anywhere any time soon.
• Google backed. Anything is possible when you Google owns you. Competition releases a good feature? YouTube has the resources to one-up it in no time. Need for more servers? YouTube will just buy a few thousand more with the $1.65Bln Google gave it. Traffic explosion? No problem – YouTube has been mooching off the Google CDN for months now.

At the end of 2007, I predicted that in a year we will experience unprecedented HD quality online video. This prediction came true when Hulu and fueled by its success CBS, ABC, NBC, and pretty much every other TV network released their free online TV sites. YouTube launched HD a few months later.

So, my YouTube prediction for 2009 is it will sign deals with major TV and movie networks to finally start showing legal TV episodes and movies. It will become the biggest legal TV and movie hub on the Internet.

10. PostgreSQL Will Gain Popularity

Sun’s buyout of MySQL in 2008 surely sent some shockwaves around. However, I predict that the following factors will contribute to PostgreSQL gaining momentum:

• certain features of MySQL were moved to Enterprise only. Open source enthusiasts don’t appreciate an open source project going partially closed source, so they will be looking for alternative software, like PostgreSQL.
• having spent years with MySQL, I am incredibly frustrated with certain quirks that should have been worked out a long time ago. As software architects look for stable, mature, cost effective, and easy to maintain databases, they will find PostgreSQL increasingly attractive.

Don’t take my word for it. I highly suggest taking a look at this whitepaper comparing MySQL and PostgreSQL. here are some highlights:

• Online operations and reorganization. This is my biggest beef with MySQL. Almost any ALTER table command will prevent writes to the table while it’s being altered. This operation requires double the table size because an ALTER simply makes a copy of the table, a rename, and then drop of the old one. This takes FOREVERRRR. PostgreSQL, on the other hand, supports a lot more online operations that will not take down the table. MySQL promised to support online ALTER TABLE by 2009. Will they keep their word? I highly doubt it.
• PostgreSQL supports function based and partial indexes.
• PostgreSQL supports nested triggers.
• PostgreSQL supports user defined datatypes.
• PostgreSQL has an IP address datatype (woot!).
• MySQL’s default engine doesn’t support online backup and recovery (*cough*, MyISAM, *cough*). Don’t even get me started on MyISAM, which doesn’t support referential integrity, transactions, of any other ACID properties. Yes, I know, there's InnoDB. It’s a lot better. But it’s still not good enough.

If you have spent time interfacing with both MySQL and PostgreSQL, I’d like to hear from you. Everyone I talked to so far who had used both, preferred PostgreSQL.

For all the MySQL fanboys, I was and still am one of you, I use MySQL every day. I’m only trying to open your eyes so you can see the rest of the world.

In fact, I think the only reason MySQL is so much more popular than Postgresql nowadays is luck and marketing by MySQL AB (hey, it sure paid off, I’m not saying anything).

So, here’s to PostgreSQL having a bright future.

Bonus

As a bonus, here is a collection of links to other interesting predictions for 2009:

• http://www.pr-squared.com/2008/12/social_media_predictions_2009.html – a great list of social media predictions.
• http://battellemedia.com/archives/004772.php – 14 online predictions as well as analysis of past years’ predictions.
• http://news.cnet.com/5-predictions-for-2009/ – 5 CNET Digital Home predictions.
• http://www.engadget.com/2009/01/01/predictions-for-2009/ – Engadget’s predictions.
• http://www.freedom-to-tinker.com/blog/felten/predictions-2009 – 38 predictions freedom and technology related predictions.
• http://www.readwriteweb.com/archives/2009_predictions_across_the_we.php – ReadWriteWeb’s predictions across the web.

• HTC Hero Coming To Sprint October 11th! $179.99, Powerful, And Sexy. Here Is Why You Need To Own It
• Sun Definitely Developing A Phone This Year
• The Real Reasons To Use Twitter (Get Over Your Prejudice Already)
• WordPress Developers – How Do You Make A Living [Poll + Discussion]?
• Meet Firefox For Mobile [Video + Feature Highlights + More Info]

Download Version 1.3 (03/22/12):

• Bash.Shortcuts.pdf for Adobe Acrobat (305.8 kB) downloaded 7308 times
• Bash.Shortcuts.docx for Word 2007+ (22.5 kB) downloaded 2733 times
• Bash.Shortcuts.doc for Word 97-2003 (52.5 kB) downloaded 1928 times

And, for quick reference, here's the ugly html version:

Hope you guys will find this list helpful. I think it is relatively complete, but feel free to add any omissions.

• Some Useful vim Commands – My vim Cheatsheet
• Essential Firefox Extensions (Plugins, Add-Ons) And Tips – A Comprehensive Guide :: Part 1 :: Tips
• Supercharge Your GNU Screen With A Power "Taskbar" And Never Feel Lost Again
• [Android] Auto Formatting Android XML Files With Eclipse
• Mass Renaming Directories And Files Using Total Commander

A short while ago, Peteris posted his very thorough experience interviewing at Google. Needless to say, the level of detail is astounding. Unfortunately, he didn’t get the job but the post is very positive and informative. Here is the gist and a short excerpt:

• There were 3 phone interviews and 5 on-site interviews.
• Peteris flew in all the way from Latvia, fully sponsored by Google. They paid for his flight, hotel, transportation, and food – brilliant!
• The interviews were very technical, with an emphasis on algorithms.
• Google’s preferred language seems to be C/C++, though I know they use Python extensively as well.
• If you’re interviewing, you better be very solid in computer science theory and data structures. At the same time, they treat every error in code very seriously, so even memory leaks are not acceptable. In fact, it looks like a few such mistakes cost Peteris his potential job.
• The questions included data structures, networking, OS/filesystem, algorithms, and past work experience.
• Google, of course, treated Peteris to a great lunch.

In order to prepare for his interviews, Peteris reviewed lots of both theoretical and practical problems. He posted a great list of books:

• TCP/IP Illustrated
• MIT’s Introduction to Algorithms + his notes on algorithms
• Building Scalable Web Sites
• C++ Cookbook
• Python Cookbook
• Perl Cookbook

to which I can myself add

• The C++ Standard Library: A Tutorial and Reference

and articles

• Google C++ Style Guide <—excellent style guide, every C++ programmer needs to read it
• Web Search for a Planet: The Google Cluster Architecture
• Algorithm Tutorials on TopCoder
• Corey Trager’s Google Interview
• Rod Hilton’s Google Interview
• Ben Watson’s Google Interview
• Shaun Boyd’s Google Interview
• How I Blew My Google Interview by Henry Blodget
• Get That Job at Google by Steve Yegge
• Tales from the Google’s interview room
• Google Interview Questions
• Google Interview Questions — Fun Brain Teasers!
• The Google File System
• Bigtable: A Distributed Storage System for Structured Data
• MapReduce: Simplified Data Processing on Large Clusters
• Failure Trends in a Large Disk Drive Population

Read the full experience at http://www.catonmat.net/blog/my-job-interview-at-google.

Thanks, Peteris and good luck on your next interview.

• Hadoop Primer – Yet Another Hadoop Introduction
• Impressions From The StackOverflow's DevDays Conference In San Francisco
• Setting Up A MySQL Cluster
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Conference Liveblogging: Introduction To The BLOB Streaming Project (Wednesday 3:00PM)

Debugging on the client side with -v didn't help much:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
...
lots of boring shit
...
debug1: Found key in /home/artem/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
...
more boring shit
...
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: /home/artem/.ssh/id_rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/artem/.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:

After breaking my head over possible reasons why the pile of junk that thinks it's smarter than me next to my feet doesn't work, kicking it a few times, and observing the same result, I turned to debugging the ssh daemon itself – sshd.

• The -d option disables the daemon mode and enables debug mode, in which only 1 connection is accepted for the lifetime of the server, after which it simply quits.
• -dd simply enables a more detailed output.
• -e switches this debug output from a log file to STDOUT.

However, to free up port 22, I had to stop the daemon that was already running, or else a "Bind to port 22 on 0.0.0.0 failed: Address already in use." error appeared (duh). An interesting question though, especially for people doing this to remote boxes, what happens when one stops sshd? Ever thought of doing that but instead ran over to your mommy crying like a little girl? Well, fear no more, because I'll tell you exactly what happens:

1. New users will have their connection refused.
2. Your own connection will not be interrrupted. sshd works by spawning a new instance of itself for every incoming connection, so your own sshd process will stay in memory.

So where was I?

1

/usr/sbin/sshd -dd -e

1
2
3
4

...
Authentication refused: bad ownership or modes for directory /home/monkey
...
Failed publickey for monkey from 192.168.1.30 port 56287 ssh2

AhA!! (emphasis on the last 'a'). What have we here?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

artem@DeathStar:~> cd /home/
artem@DeathStar:/home/> l
drwxrwx--- 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
DeathStar:/home/ # chmod 755 monkey
drwxr-xr-x 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.168.1.30 [192.168.1.30] port 22.
debug1: Connection established.
debug1: identity file /home/artem/.ssh/id_rsa type 1
debug1: identity file /home/artem/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.6
debug1: match: OpenSSH_4.6 pat OpenSSH*
...

Connection established, all systems are go, the key has been accepted.

Inspired by http://linux.derkeiler.com/Mailing-Lists/Fedora/2005-08/1105.html

P.S. Don't forget to /etc/init.d/sshd start.

• The Most Awesome VPN Tip: How To Make Windows Automatically Use Your Local WiFi/LAN Connection Directly For Requests That Don't Need To Go Through VPN
• How To Export/Import Your ExpanDrive/SFTPDrive Drives And Settings
• A Better diff Or What To Do When GNU diff Runs Out Of Memory ("diff: memory exhausted")
• Beer Planet Hosting Moved Away From Dreamhost (Thank God, Finally!)
• Youtube Custom RSS Search Results

1
2
3
4
5
6
7

rcapache2 restart
/usr/sbin/httpd2-prefork: symbol lookup error: /usr/sbin/httpd2-prefork: undefined symbol: apr_ldap_ssl_init
Starting httpd2 (prefork) /usr/sbin/httpd2-prefork: symbol lookup error: /usr/sbin/httpd2-prefork: undefined symbol: apr_ldap_ssl_init
 
The command line was:
/usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -DSSL
                                                                      failed

So I tried reinstalling libapr and apache2-utils related stuff with no luck. ldconfig didn't help either. It's not until I looked at /usr/lib and relinked a few things that it started working. What the hell, SUSE?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

/usr/lib # l libapr*
-rw-r--r-- 1 root root 239306 2007-09-21 23:41 libapr-1.a
-rw-r--r-- 1 root root    821 2007-09-21 23:41 libapr-1.la
lrwxrwxrwx 1 root root     17 2008-07-30 00:21 libapr-1.so -> libapr-1.so.0.2.9*
lrwxrwxrwx 1 root root     18 2008-07-30 00:21 libapr-1.so.0 -> libapr-1.so.0.2.12*
-rwxr-xr-x 1 root root 543841 2008-07-29 23:47 libapr-1.so.0.2.12*
-rwxr-xr-x 1 root root 148372 2007-09-21 23:41 libapr-1.so.0.2.9*
-rw-r--r-- 1 root root 156300 2007-09-23 13:33 libaprutil-1.a
-rw-r--r-- 1 root root    912 2007-09-23 13:33 libaprutil-1.la
lrwxrwxrwx 1 root root     21 2008-07-30 00:22 libaprutil-1.so -> libaprutil-1.so.0.2.9*
lrwxrwxrwx 1 root root     22 2008-07-30 00:21 libaprutil-1.so.0 -> libaprutil-1.so.0.2.12*
-rwxr-xr-x 1 root root 291520 2008-07-29 23:47 libaprutil-1.so.0.2.12*
-rwxr-xr-x 1 root root  98148 2007-09-23 13:33 libaprutil-1.so.0.2.9*
 
/usr/lib # ln -sf libaprutil-1.so.0.2.9 libaprutil-1.so.0
/usr/lib # ln -sf libapr-1.so.0.2.9 libapr-1.so.0
 
/usr/lib # l libapr*
-rw-r--r-- 1 root root 239306 2007-09-21 23:41 libapr-1.a
-rw-r--r-- 1 root root    821 2007-09-21 23:41 libapr-1.la
lrwxrwxrwx 1 root root     18 2008-07-30 00:34 libapr-1.so -> libapr-1.so.0.2.9*
lrwxrwxrwx 1 root root     18 2008-07-30 00:21 libapr-1.so.0 -> libapr-1.so.0.2.9*
-rwxr-xr-x 1 root root 543841 2008-07-29 23:47 libapr-1.so.0.2.12*
-rwxr-xr-x 1 root root 148372 2007-09-21 23:41 libapr-1.so.0.2.9*
-rw-r--r-- 1 root root 156300 2007-09-23 13:33 libaprutil-1.a
-rw-r--r-- 1 root root    912 2007-09-23 13:33 libaprutil-1.la
lrwxrwxrwx 1 root root     21 2008-07-30 00:35 libaprutil-1.so -> libaprutil-1.so.0.2.9*
lrwxrwxrwx 1 root root     21 2008-07-30 00:35 libaprutil-1.so.0 -> libaprutil-1.so.0.2.9*
-rwxr-xr-x 1 root root 291520 2008-07-29 23:47 libaprutil-1.so.0.2.12*
-rwxr-xr-x 1 root root  98148 2007-09-23 13:33 libaprutil-1.so.0.2.9*
 
/usr/lib # rcapache2 restart
Syntax OK
Shutting down httpd2 (waiting for all children to terminate)          done
Starting httpd2 (prefork)                                             done

Voila, apache starts now. Grrr, why linker, why??

• How To Make Your Site Lightning Fast* By Compressing (deflate/gzip) Your HTML, Javascript, CSS, XML, etc In Apache
• How To Resize/Grow VMware Linux Disks and Partitions
• How To Install The Latest SOAP::Lite Using Perl CPAN
• Hadoop Primer – Yet Another Hadoop Introduction
• How To View A Specific SVN Revision In Your Browser

I came up with 2, depending on how detailed the information needs to be, which I'm about to share in this post.

1. If you want exact file and directory names, you can run:

svn status -u

If any files need updating, you will see a * before the file name.

svn status wc
M     wc/bar.c
A  +   wc/qax.c

svn status -u wc
M            965    wc/bar.c
       *     965    wc/foo.c
A  +         965    wc/qax.c
Status against revision:   981

Or more verbose

svn status --show-updates --verbose wc
M            965       938 kfogel       wc/bar.c
       *     965       922 sussman      wc/foo.c
A  +         965       687 joe          wc/qax.c
             965       687 joe          wc/zig.c
Status against revision:   981

Parsing the output in Perl, for instance, should be trivial. A connection to the repository is established for this check, so be sure to catch in your code the times when such connection is not available.

2. If you only care about whether a specific directory needs to be updated or not, here's a quicker method:

svn info vs svn info -rHEAD

cd somedir;
svn info -r HEAD | grep -i "Last Changed Rev"
Last Changed Rev: 8544
svn info | grep -i "Last Changed Rev"
Last Changed Rev: 8531

If these numbers are not the same, an update is needed.

What can I be missing? Are there any other creative ways, or is svn update going to support dry-run? Feel free to leave a comment if you know something I don't.

• How To View A Specific SVN Revision In Your Browser
• How To SVN Update All Your WordPress Plugins In One Go
• Converting from CVS to SVN: Developer's Notes And Why SVN Is Better
• How To Show Hidden/Invisible Files In Total Commander, Both Locally And On An FTP Server
• How To Properly Set SVN svn:externals Property In SVN Command Line

There is a different solution, however, that is not dependent on file sizes. Enter rdiff – rsync's backbone. You can read about it here: http://en.wikipedia.org/wiki/Rsync (search for rdiff).

The upsides of rdiff are:

• with the same 4.5GB files, rdiff only ate about 66MB of RAM and scaled very well. It never crashed to date.
• it is also MUCH faster than diff.
• rdiff itself combines both diff and patch capabilities, so you can create deltas and apply them using the same program

The downsides of rdiff are:

• it's not part of standard Linux/UNIX distribution – you have to install the librsync package.
• delta files rdiff produces have a slightly different format than diff's.
• delta files are slightly larger (but not significantly enough to care).
• a slightly different approach is used when generating a delta with rdiff, which is both good and bad – 2 steps are required. The first one produces a special signature file. In the second step, a delta is created using another rdiff call (all shown below). While the 2-step process may seem annoying, it has the benefits of providing faster deltas than when using diff. In fact, you can pipe the first step into the second one without any trouble if you want, which is what I ended up doing).

Usage:

1
2
3
4
5
6
7
8
9

$ rdiff signature ORIGINAL.txt SIGNATURE.sig
 
$ l -h SIGNATURE.sig
-rw-r--r-- 1 user users 25M 2008-04-23 22:32 SIGNATURE.sig
 
$ rdiff delta SIGNATURE.sig MODIFIED.txt DELTA.rdiff
 
$ l -h DELTA.rdiff
-rw-r--r-- 1 user users 82M 2008-04-23 22:36 DELTA.rdiff

And here's what you would do to reassemble MODIFIED.txt:

1
2
3
4
5
6

$ rdiff patch ORIGINAL.txt DELTA.rdiff MODIFIED_REASSEMBLED.txt
 
$ l *.txt
-rw-r--r-- 1 user users 4,471,493,588 2008-04-23 20:24 MODIFIED.txt
-rw-r--r-- 1 user users 4,471,493,588 2008-04-23 22:44 MODIFIED_REASSEMBLED.txt
-rw-r--r-- 1 user users 4,403,302,981 2008-04-23 20:20 ORIGINAL.txt

Just as expected – everything matches.

Now, all of this could have been done in one go like this:

1

rdiff signature ORIGINAL.txt | rdiff delta -- - MODIFIED.txt DELTA.rdiff

As far as my usage of such a useful diff program, I was doing CSV dumps of certain fields from a MySQL database, like so:

1

SELECT * FROM table WHERE some_condition='1' ORDER BY id DESC INTO OUTFILE '/home/dump/dump.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"';

and then applying rdiff to get the [quite small] daily deltas.

That's all folks!

• Watch – A Useful Linux Command You May Have Never Heard Of
• How To Download And Install Windows Media Player 11 Bypassing WGA
• Navicat For MySQL Bugs Filed
• Converting from CVS to SVN: Developer's Notes And Why SVN Is Better
• How To SVN Update All Your WordPress Plugins In One Go

1

tar -tzf file.tar.gz

-t: lists files
-f: instructs tar to deal with the following filename (file.tar.gz)
-z: informs tar that the it's dealing with a gzip file (-j if it's bzip2)

• Quick Perl Snippet: Finding If A File Has A Media Extension Using Regex
• Some Useful vim Commands – My vim Cheatsheet
• Mastering The Linux Shell – Bash Shortcuts Explained (Now With Cheat Sheets)
• What A Woman Says, What She Really Means… What A Man Says, What He Really Means…
• Watch – A Useful Linux Command You May Have Never Heard Of

Passwd::Unix will corrupt your /etc/shadow file and rearrange login names and their corresponding password hashes.

The current version of Passwd::Unix corrupted my /etc/shadow upon only
calling the passwd() function. Immediately users started to report not
being able to login.

After examining the situation, I found that Passwd::Unix rearranges all
users in /etc/shadow in some way, but it only does it to the
usernames, and not the password hashes. Thus, you will get corrupted accounts. Moreover,
users are now able to login to one OTHER account, not their own,
depending on how the usernames got shuffled.

Thankfully, I had a recent backup but I definitely don’t want anyone
else to suffer.

I’m using perl 5.10, SUSE 10.3. If it’s incompatible with SUSE, it needs
to say so and exit.

I've filed the bug here: http://rt.cpan.org/Public/Bug/Display.html?id=35323.

You have been warned.

• How *Not* To Implement A Web Application That Handles External Authentication, Using BeTwittered.com As An Example
• cpan – The Perl Module Manager
• A Better diff Or What To Do When GNU diff Runs Out Of Memory ("diff: memory exhausted")
• How To Install The Latest SOAP::Lite Using Perl CPAN
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)