beer planet

Updated: June 23rd, 2009

Introduction

StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).

The Story

So, one day someone at StackOverflow started a "Hidden features of" post about a famous language (I don't feel like finding out which one was first exactly), and it turned out to be so popular that other posts in the same series started popping up.

Such questions were quickly turned into community wikis, for the purposes of harvesting and organizing information coming from the best developers on the planet and voted by users of the site. There are literally hundreds of answers, sorted by votes.

Hidden Features Of

Programming Languages

Hidden features of ASP.NET

Hidden features of C

Hidden features of C++

Hidden features of C#

Hidden features of D

Hidden features of Delphi

Hidden Features of F#

Hidden features of Java

Hidden features of JavaScript

Hidden features of Haskell

Hidden features of Lua

Hidden features of Objective-C

Hidden features of Perl

Hidden features of PHP

Hidden features of Python

Hidden features of Ruby

Hidden Features of Ruby on Rails

Hidden features of Scala

Hidden Features of TCL

Hidden features of VB.Net

Databases

Hidden features of MySQL

Hidden features of Oracle

Hidden features of PostgreSQL

Hidden features of SQL Server

Other

Hidden features of Bash - also see my bash cheatsheet.

Hidden features of CSS

Hidden features of Eclipse

Hidden features of Greasemonkey

Hidden features of HTML

Hidden features of HTTP

Hidden features of jQuery

Hidden features of mod_rewrite

Hidden Features of RegEx

Hidden features of Visual Studio (2005-2008)

I will try to maintain this list, adding new languages that join the series as I find them. Now go learn something new!

Introduction

Every time I have to deal with svn:externals in SVN, I forget the command line syntax. Every single damn time. Normally, I use SVN GUI clients, such as SmartSVN, which make it very simple to add an svn:externals property. But for command line, it always takes looking at 25 different sites on google, which are all incredibly unhelpful for this question for some reason. Trying "svn help propset" on the command line was bloated and equally useless.

So this time I needed to write it down and make sure everyone who needed help with svn:externals would find exactly what they need here. I hope this page will soon come up on top of all the unhelpful results on google for "propset svn:externals" and other related queries.

The Problem

I want to set a simple svn:externals property in one of my project's directories, lets say 'plugins' (talking about Wordpress here). The outcome would be a directory called 'akismet' within 'plugins' that points to a remote svn url.

Various combinations of trying to do it produced pathetic results, like

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

svn propset svn:externals . akismet http://plugins.svn.wordpress.org/akismet/trunk
svn: Error parsing svn:externals property on 'akismet': '.'

svn propset svn:externals akismet http://plugins.svn.wordpress.org/akismet/trunk akismet
svn: Setting property on non-local target 'http://plugins.svn.wordpress.org/akismet/trunk' needs a base revision

The Solution

Finally, thanks to this post, I found the right command:

svn propset svn:externals 'akismet http://plugins.svn.wordpress.org/akismet/trunk' .
property 'svn:externals' set on '.'

Note that dot at the end of the command and the quotes around the directory name and url.

Now commit via

svn commit

and then

svn up
Fetching external item into 'akismet' 
A    akismet/akismet.gif 
A    akismet/akismet.php 
A    akismet/readme.txt 
Updated external to revision 127962.
 
Updated to revision 16.

There, was it that hard, forum gurus and blog fiends?

Edit: in order to set multiple directory/url pairs in a single svn:externals property, you should put the individual dir/url pairs into a file (let's call it 'svn.externals'), like so

akismet http://svn.wp-plugins.org/akismet/trunk
all-in-one-seo-pack http://svn.wp-plugins.org/all-in-one-seo-pack/trunk

and then apply the property using

svn propset svn:externals -F svn.externals .

You should also just check in 'svn.externals' to easily keep track of it.

Updated: June 10th, 2009

* Lightning Fast is a blatant exaggeration. Got you to look though, didn't it?

Introduction

Whether you are a web developer or a self-hosting business owner, the only excuse for not activating compression capabilities of your web server can be that you didn't know about it. And now that you are reading this, there is no excuse left at all.

Here is how big a single page of this blog was before compression was enabled on CSS and Javascript files (computed by YSlow):

And here it is after compression:

As you see, the difference is quite substantial - almost 30% savings.

Compressing your HTML, XML, Javascript, CSS, etc pages will mean less data transferred between the server and the client which:

• reduces the bandwidth usage.
• provides faster page rendering which in turn leads to less user frustration, higher conversion rates, lower bounce rate, etc etc etc.

Compression is especially important for users with slow connections as every kilobyte of your code is that much more painful to them.

Compression can be very effective - you can easily shrink your text, code (HTML, XML, Javascript, CSS, etc) to 10% of the original size (of course, your mileage may vary). 100KB page that needs only 10KB to transfer? Sign me up!

So, before I talk about the solution, let me describe what exactly happens when compression is turned on and how it affects older browsers that don't support it.

Compression Mechanism Explained

In order for compression to work in the first place, the web server (Apache in my example) needs to support it. This is achieved by enabling one of Apache modules called mod_deflate. The server will then be able to compress the data to the DEFLATE standard using either the zlib (also known as deflate) or gzip implementations. Yeah, DEFLATE is both the standard the one of its implementations, for those confused. I know I was. This is best described in this Wikipedia article.

The following mechanism is used:

• the server with a compression extension enabled is able to serve either compressed (smaller) or uncompressed (larger) pages, depending on what the client supports.
• the client (that is, your browser) sends a special header called "Accept-Encoding" listing the DEFLATE implementations it's capable of decompressing. For example "gzip,deflate".
• the server picks the best compression supported by the client (if any), compresses the files, and sends them over to the client.
• the client receives the compressed files and decompresses them.

Are Your Pages Already Compressed? Test Them!

If you are not sure whether you are already serving compressed pages or not, test them! My favorite way is by using Charles HTTP Debugger. Another option is by downloading Firebug for Firefox and installing Yahoo's YSlow or Google's Page Speed. Just look at the response headers to see if compression is on (look for the Content-Encoding header). Here are some before and after examples:

Theme CSS

Before:

After:

jQuery

Before:

After:

Solution

Create a .htaccess file in the top directory of your site with the following contents:

1
2
3
4
5
6
7
8

# DEFLATE by type - html, text, css, xml
AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml
 
# DEFLATE by type - javascript
AddOutputFilterByType DEFLATE application/x-javascript application/javascript text/javascript text/x-js text/x-javascript
 
# DEFLATE by extension
AddOutputFilter DEFLATE js css htm html xml

Alternatively, you could put these lines into your Apache config within the Directory directive.

The AddOutputFilterByType directive adds DEFLATE filters to certain MIME types. I tried to assemble some of the common ones but feel free to add more, as each server may be configured differently and give out MIME types different from mine.

The AddOutputFilter directive binds the DEFLATE filter to specific file extensions, just in case they are not served with a proper MIME type. Feel free to add to this list as well.

Caveats

1. In order to use this whole compression/deflate/gzip business, your Apache server must first have mod_deflate enabled. Without it, you will get the HTTP 500 error (Internal Server error). You can check which mods you already have enabled by checking with the output of phpinfo() function.

In order to enable mod_deflate, uncomment the line with "deflate_module" in your Apache config file. The location of this config file is highly dependant on your system. Some examples include

• /etc/apache2/httpd.conf
• /etc/httpd/conf/httpd.conf
• c:\wamp\bin\apache\Apache2.2.11\conf\httpd.conf
• some other place where your system stores Apache config files (read the special note below for OpenSUSE).

Here's what you should have:

LoadModule deflate_module modules/mod_deflate.so

On OpenSUSE, you actually enable modules a bit differently. Go to /etc/sysconfig/apache2 and look for APACHE_MODULES=. Then add "deflate" to the list, if it's not already there.

Now, restart Apache and check the output of phpinfo() again.

2. Adding AddOutputFilter and AddOutputFilterByType to .htaccess requires such overrides to be authorized by the main Apache configuration for that directory, otherwise it will return error 500 as well. The option you are looking for is called "AllowOverride" and mine was set to "AllowOverride AuthConfig" which wasn't enough. Changing it to

AllowOverride AuthConfig FileInfo

or just

AllowOverride All

fixes the problem. You can find more info about AllowOverride here.

3. In Wordpress, if you are using Google Gears (Turbo mode) for caching some core Wordpress files, they will not show up compressed. That is because they're not served by the remote server but rather reside locally (think of it as permanent cache). I was very confused at first when I didn't see jQuery.js in the HTTP log and YSlow reported it uncompressed.

Are you a Wordpress user?

If you are a Wordpress user, don't assume Wordpress is going to automatically compress your pages. In fact, as you install more and more plugins, the payload becomes larger and larger with those additional CSS and Javascript files.

You owe it to yourself and to your users to immediately enable compression on your blog.

Here is what happened after I enabled compression on this blog.

Before:

After:

Bonus - WP Minify

For even better results, I suggest you have a look at my good friend and talented Wordpress master Thaya's plugin called WP Minify. It preprocesses and aggregates all or most of your CSS and Javascript into just 2 files, thus saving on the number of HTTP requests. It also minifies content to achieve smaller size.

My blog before WP Minify:

After WP Minify:

That's all folks. Let me know if something was unclear and I'll be glad to clarify it.

A few references that pointed me in the right direction and allowed me to provide a more complete solution:

• http://brightscape.net/compress-jquery-even-further/
• http://brightscape.net/compress-your-web-pages-with-mod_deflate/

The Problem

I use a lot of extensions. A LOT. They slow down my Firefox while giving something useful in return. Most of them are harmless and do not affect the speed too much but there are select few that are just CPU, memory, and performance hogs.

Until Firefox gets an extension manager that can show what the impact from each extension on time/CPU/memory is, one can resort to guessing, disabling, testing, and looking for clues to find these conniving little bastards.

Anyway, so where was I? Recently, my Firefox became increasingly unresponsive, especially when switching tabs. After some time, it was a pain to switch tabs altogether, so I had to resort to restarting the browser, only to find the same problem coming back again and again. Today, I finally got a clue - a bunch of such messages came on mentioning this unresponsive sript and prompting me to stop it:

Firecookie (0.8 at the time of this writing). Arrr!

Quickly, I googled the problem and sure enough - other people are suffering from the same problem.

Firecookie adds support for managing cookies directly from within Firebug and is quite handy. However, not handy enough to kill my browser on a daily basis, even with cookie handling turned off within Firebug itself.

The Result

I am pleased to report that after uninstalling Firecookie, my Firefox is noticeably faster switching tabs, does not hang, and does not use as much CPU. I do not know what the developer of Firecookie is going to do about it as he does not seem to be able to reproduce the issue. Until then, goodbye frustration and hello… whatever other extensions handle cookies.

Introduction

A question I recently saw on Stack Overflow titled Faster way to delete matching [database] rows? prompted me to organize my thoughts and observations on the subject and quickly jot them down here.

Here is the brief description of the task: say, you have 2 MySQL tables a and b. The tables contain the same type of data, for example log entries. Now you want to delete all or a subset of the entries in table a that exist in table b.

Solutions Suggested By Others

DELETE FROM a WHERE EXISTS (SELECT b.id FROM b WHERE b.id = a.id);

DELETE a FROM a INNER JOIN b on a.id=b.id;

DELETE FROM a WHERE id IN (SELECT id FROM b)

The Problem With Suggested Solutions

Solutions above are all fine if the tables are quite small and the SELECT/JOIN is fast. However, in large scale situations with replication, these queries could potentially lock up the tables from writes and severely aggravate slave lag because, as I mentioned in the MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It post, replication is single-threaded.

Thus, if a single UPDATE/DELETE query takes a considerable amount of time, when it propagates to the slaves, they will be stuck executing it and doing nothing else, lagging behind more and more.

My Thoughts And Solution

I have personally dealt with having to delete many rows from one table that exist in another and in my experience it's best to do the following, especially if you expect lots of rows to be deleted. This technique most importantly will improve replication slave lag.

So, here it is: do a SELECT first, as a separate query, remembering the IDs returned in your script/application, then continue on deleting in batches (say, 50,000 rows at a time). This will achieve the following:

• each one of the delete statements will not lock the table for too long, thus not letting replication lag get out of control. It is especially important if you rely on your replication to provide you relatively up-to-date data. The benefit of using batches is that if you find that each DELETE query still takes too long, you can adjust it to be smaller without touching any DB structures.
• another benefit of using a separate SELECT is that the SELECT itself might take a long time to run, especially if it can't for whatever reason use the best DB indexes. If the SELECT is inner to a DELETE, when the whole statement migrates to the slaves, it will have to do the SELECT all over again, potentially lagging the slaves because of how long that SELECT will take. If you use a separate SELECT query, this problem goes away, as all you're passing to the DELETE query is a list of IDs.

Do you have another opinion or see a fault with my logic? Feel free to share in the comments.

P.S. One thing to be careful about is, of course, potential edits to the table between the times the SELECT finishes and DELETEs start. I will let you handle such details by using transactions and/or logic pertinent to your application.