beer planet

Introduction

Every time I have to deal with svn:externals in SVN, I forget the command line syntax. Every single damn time. Normally, I use SVN GUI clients, such as SmartSVN, which make it very simple to add an svn:externals property. But for command line, it always takes looking at 25 different sites on google, which are all incredibly unhelpful for this question for some reason. Trying "svn help propset" on the command line was bloated and equally useless.

So this time I needed to write it down and make sure everyone who needed help with svn:externals would find exactly what they need here. I hope this page will soon come up on top of all the unhelpful results on google for "propset svn:externals" and other related queries.

The Problem

I want to set a simple svn:externals property in one of my project's directories, lets say 'plugins' (talking about Wordpress here). The outcome would be a directory called 'akismet' within 'plugins' that points to a remote svn url.

Various combinations of trying to do it produced pathetic results, like

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

svn propset svn:externals . akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Error parsing svn:externals property on 'akismet': '.'

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk akismet
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

The Solution

Finally, thanks to this post, I found the right command:

svn propset svn:externals 'akismet http://plugins.svn.wordpress.org/akismet/trunk' .
property 'svn:externals' set on '.'

Note that dot at the end of the command and the quotes around the directory name and url.

Now commit via

svn commit

and then

svn up
Fetching external item into 'akismet' 
A    akismet/akismet.gif 
A    akismet/akismet.php 
A    akismet/readme.txt 
Updated external to revision 127962.
 
Updated to revision 16.

There, was it that hard, forum gurus and blog fiends?

Edit: in order to set multiple directory/url pairs in a single svn:externals property, you should put the individual dir/url pairs into a file (let's call it 'svn.externals'), like so

akismet http://svn.wp-plugins.org/akismet/trunk
all-in-one-seo-pack http://svn.wp-plugins.org/all-in-one-seo-pack/trunk

and then apply the property using

svn propset svn:externals -F svn.externals .

You should also just check in 'svn.externals' to easily keep track of it.

Updated: June 10th, 2009

* Lightning Fast is a blatant exaggeration. Got you to look though, didn't it?

Introduction

Whether you are a web developer or a self-hosting business owner, the only excuse for not activating compression capabilities of your web server can be that you didn't know about it. And now that you are reading this, there is no excuse left at all.

Here is how big a single page of this blog was before compression was enabled on CSS and Javascript files (computed by YSlow):

And here it is after compression:

As you see, the difference is quite substantial - almost 30% savings.

Compressing your HTML, XML, Javascript, CSS, etc pages will mean less data transferred between the server and the client which:

• reduces the bandwidth usage.
• provides faster page rendering which in turn leads to less user frustration, higher conversion rates, lower bounce rate, etc etc etc.

Compression is especially important for users with slow connections as every kilobyte of your code is that much more painful to them.

Compression can be very effective - you can easily shrink your text, code (HTML, XML, Javascript, CSS, etc) to 10% of the original size (of course, your mileage may vary). 100KB page that needs only 10KB to transfer? Sign me up!

So, before I talk about the solution, let me describe what exactly happens when compression is turned on and how it affects older browsers that don't support it.

Compression Mechanism Explained

In order for compression to work in the first place, the web server (Apache in my example) needs to support it. This is achieved by enabling one of Apache modules called mod_deflate. The server will then be able to compress the data to the DEFLATE standard using either the zlib (also known as deflate) or gzip implementations. Yeah, DEFLATE is both the standard the one of its implementations, for those confused. I know I was. This is best described in this Wikipedia article.

The following mechanism is used:

• the server with a compression extension enabled is able to serve either compressed (smaller) or uncompressed (larger) pages, depending on what the client supports.
• the client (that is, your browser) sends a special header called "Accept-Encoding" listing the DEFLATE implementations it's capable of decompressing. For example "gzip,deflate".
• the server picks the best compression supported by the client (if any), compresses the files, and sends them over to the client.
• the client receives the compressed files and decompresses them.

Are Your Pages Already Compressed? Test Them!

If you are not sure whether you are already serving compressed pages or not, test them! My favorite way is by using Charles HTTP Debugger. Another option is by downloading Firebug for Firefox and installing Yahoo's YSlow or Google's Page Speed. Just look at the response headers to see if compression is on (look for the Content-Encoding header). Here are some before and after examples:

Theme CSS

Before:

After:

jQuery

Before:

After:

Solution

Create a .htaccess file in the top directory of your site with the following contents:

1
2
3
4
5
6
7
8

# DEFLATE by type - html, text, css, xml
AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml
 
# DEFLATE by type - javascript
AddOutputFilterByType DEFLATE application/x-javascript application/javascript text/javascript text/x-js text/x-javascript
 
# DEFLATE by extension
AddOutputFilter DEFLATE js css htm html xml

Alternatively, you could put these lines into your Apache config within the Directory directive.

The AddOutputFilterByType directive adds DEFLATE filters to certain MIME types. I tried to assemble some of the common ones but feel free to add more, as each server may be configured differently and give out MIME types different from mine.

The AddOutputFilter directive binds the DEFLATE filter to specific file extensions, just in case they are not served with a proper MIME type. Feel free to add to this list as well.

Caveats

1. In order to use this whole compression/deflate/gzip business, your Apache server must first have mod_deflate enabled. Without it, you will get the HTTP 500 error (Internal Server error). You can check which mods you already have enabled by checking with the output of phpinfo() function.

In order to enable mod_deflate, uncomment the line with "deflate_module" in your Apache config file. The location of this config file is highly dependant on your system. Some examples include

• /etc/apache2/httpd.conf
• /etc/httpd/conf/httpd.conf
• c:\wamp\bin\apache\Apache2.2.11\conf\httpd.conf
• some other place where your system stores Apache config files (read the special note below for OpenSUSE).

Here's what you should have:

LoadModule deflate_module modules/mod_deflate.so

On OpenSUSE, you actually enable modules a bit differently. Go to /etc/sysconfig/apache2 and look for APACHE_MODULES=. Then add "deflate" to the list, if it's not already there.

Now, restart Apache and check the output of phpinfo() again.

2. Adding AddOutputFilter and AddOutputFilterByType to .htaccess requires such overrides to be authorized by the main Apache configuration for that directory, otherwise it will return error 500 as well. The option you are looking for is called "AllowOverride" and mine was set to "AllowOverride AuthConfig" which wasn't enough. Changing it to

AllowOverride AuthConfig FileInfo

or just

AllowOverride All

fixes the problem. You can find more info about AllowOverride here.

3. In Wordpress, if you are using Google Gears (Turbo mode) for caching some core Wordpress files, they will not show up compressed. That is because they're not served by the remote server but rather reside locally (think of it as permanent cache). I was very confused at first when I didn't see jQuery.js in the HTTP log and YSlow reported it uncompressed.

Are you a Wordpress user?

If you are a Wordpress user, don't assume Wordpress is going to automatically compress your pages. In fact, as you install more and more plugins, the payload becomes larger and larger with those additional CSS and Javascript files.

You owe it to yourself and to your users to immediately enable compression on your blog.

Here is what happened after I enabled compression on this blog.

Before:

After:

Bonus - WP Minify

For even better results, I suggest you have a look at my good friend and talented Wordpress master Thaya's plugin called WP Minify. It preprocesses and aggregates all or most of your CSS and Javascript into just 2 files, thus saving on the number of HTTP requests. It also minifies content to achieve smaller size.

My blog before WP Minify:

After WP Minify:

That's all folks. Let me know if something was unclear and I'll be glad to clarify it.

A few references that pointed me in the right direction and allowed me to provide a more complete solution:

• http://brightscape.net/compress-jquery-even-further/
• http://brightscape.net/compress-your-web-pages-with-mod_deflate/

Updated: June 9th, 2009

No matter what programming language you use, there comes a time when you need to search for a file somewhere on the file system. Here, I want to talk about accomplishing this task in Perl. There are many ways of doing so, most of them boring, but I want to discuss the fun and elegant way - using File::Find::Rule.

Let me briefly discuss some of the other methods first.

Limited

Using glob() (or <>, TODO verify) you can find files in a single directory, using only the limited shell wildcard support. For example,

1

my @files = glob("tmp*");

Boring

File::Find is the de facto standard for searching in Perl.

This method finds files that end in .pl in "." and "../SomeDir", following symlinks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/perl -w
 
use File::Find;
use Data::Dumper;
use File::Basename;
my @directories_to_search = (".", "../SomeDir");
my @file_list = ();
 
find(
  { wanted =>
    sub {
      if ( basename($File::Find::name) =~ /\.pl$/i )
      { 
        push @file_list, $File::Find::name;
      }
    },
    follow => 1
  },
  @directories_to_search
);
print Dumper @file_list;

It works fine, except it's horribly ugly and boring. Let's have a look at something more fun.

The Fun And Elegant Way

File::Find::Rule. Just have a look at this beauty.

Just like above, find all .pl files in "." and "../SomeDir", following symlinks:

1
2

print Dumper (File::Find::Rule->name("*.pl")->file->extras({ follow => 1 })->
in(".", "../SomeDir"));

Same as above, except bypass .svn directories (shaves off a ton of time with a lot of directories):

1
2

print Dumper (File::Find::Rule->not(File::Find::Rule->directory->name('.svn')->
prune->discard)->name("*.pl")->file->extras({ follow => 1 })->in(".", "../SomeDir"));

Find all .log files that are older than 24 hours in "."

1
2
3

my $epoch_time_1_day_ago = time() - 60*60*24;
print Dumper (File::Find::Rule->file->name("*.log")->
mtime("<$epoch_time_1_day_ago")->in('.'));

Be sure to read the File::Find::Rule perldoc for more options and remember: have fun with your code!

Thanks to Perlbuzz and Andy Lester for pointing me to this library a few months ago.

Updated: June 9th, 2009

Introduction

If you, like me, are building or thinking of implementing a MySQL-powered application that has any need for prioritizing selecting certain data over other data, this article is for you.

Example

As a real world example, consider a queue-like video processing system. Your application receives new videos and processes them. The volume of incoming videos can at times be higher than the processing rate because the process is CPU bound, so occasionally a pretty long queue may form. You will try to process them as fast as you can but…

…suddenly, you need to process a video somewhere in the middle of the queue or an important video enters and needs immediate attention. What do you do?

An obvious solution is implementing a simple priority system where each item has a numeric priority field. Now you can sort first by priority from highest to lowest and then by id within the highest priority. Important and urgent items get a their priority changed to something higher and get processed first. There is only one problem.

Problem

The problem is pretty serious – let’s take a look at the SELECT statement. Before selecting, I’ve added 19 random rows to have some data to work on.

SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;

What kind of index would you put on this table to speed up this query? You do want to add a proper index, don’t you? DO YOU? Ok, good.

Here’s what happens without any indexes:

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
|  1 | SIMPLE      | queue | ALL  | NULL          | NULL | NULL    | NULL |   19 | Using filesort | 
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)

Using filesort, ugh, of course, due to sorting without an index.

Let’s see, how about a combined index on (priority, id)?

mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority`, `id`);
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort | 
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

Better because an index is being used but not very good because filesort is still present. “Of course!”, you slap yourself on the forehead. The first ORDER BY uses a DESCENDING order, and our key is in ASCENDING order.

So, let’s add the proper key with the right ordering instead.

mysql> ALTER TABLE `queue` DROP INDEX `priority_id`;
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority` DESC, `id`);
Query OK, 19 rows affected (0.06 sec)
Records: 19  Duplicates: 0  Warnings: 0

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort | 
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

What the deuce? This is the same result as with the previous index. Time to dig up the documentation.

Here is what the MySQL manual has to say under the ORDER BY optimization section:

MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause … if you mix ASC and DESC:

SELECT * FROM t1 ORDER BY key_part1 DESC, key_part2 ASC;

Moreover, to confuse the user even more, the index creation command accepts the DESC instruction, without actually honoring it, as specified in the CREATE INDEX section:

An index_col_name specification can end with ASC or DESC. These keywords are allowed for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

So, after so many years MySQL still doesn’t support such basic functionality – you are either stuck with a query that uses filesort or have to look for a workaround.

Solution

Since it’s not possible to mix order directions, the solution is then to change the meaning of the priority column to match your needs. Thus, in the new approach priority 1 is higher than priority 10, and the application logic needs to accommodate to that. If you caught this while the application is still young, the code may be easy to change, but otherwise it could be a major pain in the butt.

Conclusion

The moral here is: plan your queries ahead and don’t mix and match DESC and ASC ordering as MySQL will not be able to use an index to resolve it. Do it even sooner if you’re putting lots and lots of data into your tables.

Updated: June 9th, 2009

Problem

While writing a 1093985th Perl script the other day I was facing the following dilemma:

• Let’s say there is a local library, called TheUberLib.pm. It is so uber that most of my scripts, located all over the machine, include it.
• Now, let’s also say that there’s an even more uberly important binary called run_me_now_or_you_will_die but the only way to find it is by using a relative path to the aforementioned TheUberLib.pm, for example ../bin (RELATIVE TO TheUberLib.pm).
• I don’t want to hardcode the path to run_me_now_or_you_will_die because it can be different on multiple machines and the code wouldn’t be robust enough – all I know is that the path is relative to an included library.

So how does one get the path to an included TheUberLib.pm (in my case to deduce the path to run_me_now_or_you_will_die), regardless of the script location and the current directory?

I started looking at all possible ways of getting this information.

• At first I looked at $0. $0 returns the path to the Perl script itself (LibraryPath/lib_path.pl) and is completely useless here.
• The FindBin module that I sometimes use was also of no help as it also deals with the caller script path.

Solution

After digging around for a bit, this is the method I came up with that did exactly what I wanted:

1
2

use File::Basename;
my $path_to_uber_lib = dirname($INC{'TheUberLib.pm'});

and the clean way to get the final path to run_me_now_or_you_will_die is:

1
2
3

my $relative_path_to_bin_dir = "../the_path_we_want/run_me_now_or_you_will_die";
my $path_to_deadly_bin = Cwd::realpath("$path_to_uber_lib/$relative_path_to_bin_dir");
print "The path to deadly bin is: $path_to_deadly_bin\n";

which prints /tmp/LibraryPath/some_path1/the_path_we_want/run_me_now_or_you_will_die.

Here is the full program with all directories that you can check out from SVN:

svn co http://beerpla.net/svn/public/Perl/LibraryPath/

or just browse the code here: http://beerpla.net/svn/public/Perl/LibraryPath/

Explanation

The %INC hash contains all included libraries, with library names as keys and full paths as values.

The dirname() function is part of File::Basename and returns just the directory part of the path.

Finally, Cwd::realpath() is a function that resolves relative and symbolic links to canonical absolute ones.

And there you have it.