beer planet

One thing that still springs to mind when I think of the MySQL User Conference last week is Sun's opening keynote. While talking about Sun's market penetration with open source software, Jonathan Schwartz, Sun's CEO, slipped in a short mention of the mobile market saying something along the lines of "Sun is going to be entering the mobile market later on this year". He didn't spend more than 5 seconds talking about it, moving on to the acquisition of MySQL.

Last year, Sun already made an announcement of JavaFX, a Java-based mobile platform but didn't provide any concrete timelines, so I was excited to hear the more on the subject. With Apple iPhone's advent last year and Google entering the same space later on this year with Android, Sun's addition to the game definitely won't hurt consumers. After all, competition usually leads to better products.

So when is the phone coming, Jonathan? My guess is you're going to try as hard as possible to compete with Google's second half of 2008 timeline but are you going to manage to beat it? And will it blow us away? I guess time will show.

Updated: March 29th, 2008

This article contains my notes and detailed instructions on setting up a MySQL cluster. After reading it, you should have a good understanding of what a MySQL cluster is capable of, how and why it works, and how to set one of these bad boys up. Note that I'm primarily a developer, with an interest in systems administration but I think that every developer should be able to understand and set up a MySQL cluster, at least to make the dev environment more robust.

Notes

• In short, a MySQL cluster allows a user to set up a MySQL database shared between a number of machines. Here are some benefits:
  • High availability. If one or some of the machines go down, the cluster will stay up, as long as there is at least one copy of all data still present. The more redundant copies of data there are, the more machines you can afford to lose.
  • Scalability. Distributed architecture allows for load balancing. If your MySQL database is getting hit with lots of queries, consider setting up a cluster to spread this load in almost linear fashion. A 4 node cluster should be able to handle twice as many queries as a 2 node cluster.
  • Online backups.
  • Full support for transactions.
• Must-have manual: MySQL Clustering by Alex Davies and Harrison Fisk, MySQL Press.
• First and foremost, I would like to get this out of the way (from MySQL Clustering):
  • Response time with MySQL Cluster is quite commonly worse than it is with the traditional setup. Yes, response time is quite commonly worse with clustering than with a normal system. If you consider the architecture of MySQL Cluster, this will begin to make more sense.
  • When you do a query with a cluster, it has to first go to the MySQL server, and then it goes to storage nodes and sends the data back the same way. When you do a query on a normal system, all access is done within the MySQL server itself. It is clearly faster to access local resources than to read the same thing across a network. Response time is very much dependant on network latency because of the extra network traffic. Some queries may be faster than others due to the parallel scanning that is possible, but you cannot expect all queries to have a better response time.
  • So if the response time is worse, why would you use a cluster? First, response time isn't normally very important. For the vast majority of applications, 10ms versus 15ms isn't considered a big difference.
  • Where MySQL Cluster shines is in relation to the other two metrics: throughput and scalability.
• A typical MySQL cluster setup involves 3 components in at least this configuration:
  • 1 management (ndb_mgmd) node.
    • Management nodes contain the cluster configuration.
    • A management node is only needed to connect new storage and query nodes to the cluster and do some arbitration.
    • Existing storage and query nodes continue to operate normally if the management node goes down.
    • Therefore, it's relatively safe to have only 1 management node running on a very low spec machine (configuring 2 management nodes is possible but is slightly more complex and less dynamic).
    • Interfacing with a management node is done via an ndb_mgm utility.
    • Management nodes are configured using config.ini.
    • My setup here involves 1 management node.
  • 2 storage (ndbd) nodes.
    • You do not interface directly with those nodes, instead you go through SQL nodes, described next.
    • It is possible to have more storage nodes than SQL nodes.
    • It is possible to host storage nodes on the same machines as SQL nodes.
    • It is possible, although not recommended, to host storage nodes on the same machines as management nodes.
    • Storage nodes will split up the data between themselves automatically. For example, if you want to store each row on 2 machines for redundancy (NoOfReplicas=2) and you have 6 storage nodes, your data is going to be split up into 3 distinct non-intersecting chunks, called node groups.
    • Given a correctly formulated query, it is possible to make MySQL scan all 3 chunks in parallel, thus returning the result set quicker.
    • Node groups are formed implicitly, meaning you cannot assign a storage node to a specific node group. What you can do, however, is manipulate the IDs of the nodes in such a way that the servers you want will get assigned to the node groups you want. The nodes having consecutive IDs get assigned to the same node group until there are NoOfReplicas nodes in a node group, at which point a node group starts.
    • Storage nodes are configured using /etc/my.cnf. They are also affected by settings in config.ini on the management node.
    • My setup here involves 4 storage nodes.
  • 2 query (SQL) nodes.
    • SQL nodes are regular mysqld processes that access data in the cluster. You guessed it right - the data sits in storage nodes, and SQL nodes just serve as gateways to them.
    • Your application will connect to these SQL node IPs and will have no knowledge of storage nodes.
    • It is possible to have more SQL nodes than storage nodes.
    • It is possible to host SQL nodes on the same machines as storage nodes.
    • It is possible, although not recommended, to host SQL nodes on the same machines as management nodes.
    • SQL nodes are configured using /etc/my.cnf. They are also affected by settings in config.ini on the management node.
    • My setup here involves 4 SQL nodes.
• Normally a cluster doesn't want to start if not all the storage nodes are connected (from MySQL Clustering).
  • Therefore, the cluster waits longer during the restart if the nodes aren't all connected so that the other storage nodes can connect. This period of time is specified in the setting StartPartialTimeout, which defaults to 30 seconds. If at the end of 30 seconds, a cluster is possible (that is, it has one node from each node group) and it can't be in a network partitioned situation (that is, it has all of one node group), the cluster will perform a partial cluster restart, in which it starts up even though storage nodes are missing.
  • If the cluster is in a potential network partitioned setup, where it doesn't have all of a single node group, then it will wait even longer, with a setting called StartPartitionedTimeout, which defaults to 60 seconds.
• Adding databases propagates to all SQL nodes (at least with the latest version of MySQL), so when you create a new database, you only need to do it once on any SQL node. However, users dont propagate, so each SQL node will need to have its own users set up. Warning: do NOT try to change the MySQL internal tables (the ones in database mysql) to type ndbcluster as the cluster will break.
• I will think of something else to put here.

My Setup

This is my sample configuration with sample IPs:

• mysql-5.1.22-rc-linux-i686-icc-glibc23
• 1x management node (OpenSUSE): 10.0.0.1
• 4x storage (ndbd) nodes (OpenSUSE): 10.0.0.2, 10.0.0.3, 10.0.0.4, 10.0.0.5.
• 4x query (SQL) nodes (OpenSUSE): 10.0.0.2, 10.0.0.3, 10.0.0.4, 10.0.0.5.
• NoOfReplicas = 2, meaning there will be 2 copies of all data and therefore 4/2=2 node groups.
• Cluster data will sit in /var/lib/mysql-cluster.

Sample Screenshot

Here is a sample screenshot of another one of my configurations showing a similar setup. This is the output of show on the management node:

Setup Instructions

On the management node (as root):

1
2
3
4
5
6
7

groupadd mysql
useradd -g mysql mysql
mkdir -p /root/src/
cd /root/src/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.22-rc-linux-i686-icc-glibc23.tar.gz/from/http://mysql.he.net/
tar xvzf mysql-*.tar.gz
rm mysql-*.tar.gz

• ndb_mgmd is the management server
• ndb_mgm is the management client

1
2
3
4
5

cp mysql-*/bin/ndb_mg* /usr/bin/
chmod +x /usr/bin/ndb_mg*
mkdir /var/lib/mysql-cluster
chown mysql:mysql /var/lib/mysql-cluster
vi /var/lib/mysql-cluster/config.ini

Download /var/lib/mysql-cluster/config.ini

1
2
3

ndb_mgmd -f /var/lib/mysql-cluster/config.ini
ndb_mgm
show

On each storage and SQL node (as root):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

groupadd mysql
useradd -g mysql mysql
cd /usr/local
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.22-rc-linux-i686-icc-glibc23.tar.gz/from/http://mysql.he.net/
tar xvzf mysql-*.tar.gz
rm mysql-*.tar.gz
ln -s `echo mysql-*` mysql
cd mysql
chown -R root .
chown -R mysql data
chgrp -R mysql .
scripts/mysql_install_db --user=mysql
cp support-files/mysql.server /etc/init.d/
chmod +x /etc/init.d/mysql.server
vi /etc/my.cnf

Download /etc/my.cnf

1
2
3
4
5
6
7
8
9

mkdir /var/lib/mysql-cluster
chown mysql:mysql /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
su mysql
/usr/local/mysql/bin/ndbd --initial # start the storage node and force it to (re)read the config
exit
echo "/usr/local/mysql/bin/ndbd" > /etc/init.d/ndbd
chmod +x /etc/init.d/ndbd
/etc/init.d/mysql.server restart # start the query node

SUSE:

1
2
3
4

chkconfig --add mysql.server # this is SUSE's way of starting applications on system boot
chkconfig --add ndbd
chkconfig --list mysql.server
chkconfig --list ndbd

Ubuntu:

1
2
3
4
5

sudo apt-get install sysv-rc-conf # this is chkconfig's equivalent in Ubuntu
sysv-rc-conf mysql.server on
sysv-rc-conf ndbd on
sysv-rc-conf --list mysql.server
sysv-rc-conf --list ndbd

That's it! At this point you should go back to the management console that you logged into earlier (ndb_mgm) and issue the 'show' command again. If everything is fine, you should see your data and SQL nodes connected. Now you can login to any SQL node, make some users, and create new ndb tables. If you're experiencing problems, do leave a message in the comments.

In the next mysql cluster article, I will explore various cluster error messages I have encountered as well as config file tweaking. Now go and spend some time outside in the sun - life is too short to waste it at a dark office.

Updated: March 26th, 2008

April 14-17th is going to be an exciting time. Why? Because the 2008 MySQL Conference and Expo is going to be held in Santa Clara, CA. Who would want to miss out on a chance to lurk around, let alone talk to, some of the smartest people in the MySQL world? Well, those who don't have at least $1000+, of course. A 3 day pass to the conference without tutorials costs a whopping $1199. A full pass would dry up your pockets $1499.

Well, "good news everyone". Thanks to Sheeri Cabral of The Pythian Group, PlanetMySQL.org, Jeremy, and, most importantly, LinuxQuestions.org, I am now in possession of a 3-day conference pass!! I'm incredibly excited that I will be able to attend and finally meet many geniuses, including the ones mentioned on my Must-Know People In The MySQL Field page. I've never won anything worth over 50 cents before. As a funny side note, there were 4 pages of replies to the raffle post, 90% of them saying that they could not attend (mostly due to living in other countries), so in reality only 2-3 people out of everyone could actually attend. I like those odds.

Here is a link to the oh so colorful conference schedule. I'm particularly interested in this short list of highly exciting subjects:

April 15th

• State of MySQL - the keynote by Mårten Mickos (former CEO of MySQL).
• Performance Guide for MySQL Cluster - parallel query processing guide perhaps?
• Lessons Learned in Building a Highly Scalable MySQL Database - definitely need more lessons.
• Big Bird (Scaling Twitter) - should be fun, considering I heard twitter was started in ruby on rails and had major problems scaling out.
• InnoDB: Status, Architecture, and New Features - an update would be nice, kthx.
• Dramatically Improving MySQL Database Performance in Data Warehouse Applications - more lessons!
• Investigating Innodb Scalability Limits by Peter Zaitsev from MySQL Performance Blog!
• Disaster is Inevitable—Are You Prepared? by Farhan Mashraqi from Fotolog.
• Mitigating Replication Latency in a Distributed Application Environment - I need to get rid of these constant replication lags already, for the love of god!

April 16th

• Portable Scale-out Benchmarks for MySQL - tasty MySQL bench goodies? Sign me up.
• Applied Partitioning and Scaling Your Database System - sharding, disk spanning? Whatever this will be, I'm all ears.
• Architecture of Maria: A New Storage Engine with a Transactional Design - I'm very interested in this new, very promising future replacement engine for MyISAM.
• Astronomy, Petabytes, and MySQL - sounds like fun. I liked astronomy… I think.
• Benchmarking and Monitoring: Tools of the Trade (Part I) - more tools, invaluable.
• Benchmarking and Monitoring: Tools of the Trade (Part II) - even more? Tool overload for today.

April 17th

• A Match Made in Heaven? The Social Graph and the Database - by Jeff Rothschild from Facebook.com, sounds like something to wake me up in the morning.
• Stored Routines: Tips, Tricks, and Solutions
• MySQL Proxy, the Friendly Man in the Middle - probably one of the most interesting things in development right now.
• Sphinx: High Performance Full Text Search for MySQL
• Helping InnoDB Scale on Servers with Many CPU Cores and Disks - more scaling, always good.
• MySQL Hidden Treasures - man, how do I combine 3 sessions that go on at the same time?
• Top 20 DB Design Tips Every Architect Needs to Know
• Partitioned mySQL and …. realtime - by Dathan Pattishall from Flickr.
• Deadly Sins Using MySQL and PHP - by Arjen Lentz

April 18th

• temporarily unload all the information gathered in the previous 3 days and drown in beer

Excitement is in the air. Can you FEEL IT?

If you're like me, you constantly move and rename files and directories. You are also an extremely productive person with evangelical ideals of making every task as efficient as it can be. In this tutorial, I will use my favorite must-have file manager called Total Commander (formerly, Windows Commander) and its brilliant Multi-Rename Tool.

You can download a shareware version of Total Commander at www.ghisler.com. I encourage you to buy it after you try it as it'll soon become an integral part of your life. I've been using it for more than 10 years now and seriously can't imagine my computer without it.

Now for some quick tasks I'd like to accomplish using the Multi-Rename Tool in under a minute each that would otherwise take me ages (also being quite boring and tedious). In my example, I'm going to use 5 files, but feel free to extend it to any number - multi is multi after all. The tool supports regular expressions (regex) to execute more advanced operations.

And the sample files are:

vac-sick-leave request form 03-10-08.xls
vac-sick-leave request form 05-30-07.xls
vac-sick-leave request form 10-16-07.xls
vac-sick-leave request form 11-22-07.xls
vac-sick-leave request form 12-19-07.xls

I fire up Total Commander, highlight the files using the gray *, right clicking and dragging on the names, or pressing Insert - the possibilities are endless, and fire up the Multi-Rename Tool from the Files menu (or just Ctrl-M). Here's what the screen looks like now:

Now for some tasks. I find it better to show the capabilities of this tool in examples, rather than boring descriptions. You can find a full manual in Appendix A.

Task #1. Rename all files replacing 'vac' with 'vacation'.

Fill the boxes as follows:

Search for: vac
Replace with: vacation

Notice how the New name column changes instantly as you type to show the resulting file names.

Now you can press Start to make the changes final. Have no fear: there's an Undo button that will undo the last rename operation in case you made an error, even if you close and reopen the Multi-Rename Tool. You can also press the little button to the left of Start, which will reload the newly renamed filenames into the tool for further manipulations.

This solution is going to work fine in this case, but it's not going to work for all possible cases. For example, if the file name were vac-sick-evacuation request form 03-10-08.xls, the resulting filename would be vacation-sick-evacationuation request form 03-10-08.xls. In order to fix this, I will use a simple regular expression:

Search for: ^vac\b
Replace with: vacation
RegEx: check

^ would match only if vac is in the beginning of the filename and \b signifies a word boundary, meaning it wouldn't match vaca or vac3, but would match vac- or vac$. You can find the information about regex from the Total Commander manual in Appendix B at the end of this article.

Similarly, to replace all spaces with dots (replacing consecutive spaces with just 1 dot), you can enter:

Search for: \s+
Replace with: .
RegEx: check

\s - the space character, followed by +, will match 1 or more consecutive spaces.

Task #2. Change dates from US format MM-DD-YY to YYYY-MM-DD.

Search for: (\d{2})\-(\d{2})\-(\d{2})
Replace with: 20$3-$1-$2
RegEx: check

\d will match a digit, \d{2} will only match when 2 consecutive digits are seen, the dash symbol doesn't need to be escaped with \ in this case but it's usually a good practice to do so because it has a special meaning in regex. Finally, the parentheses signify that I want to remember what I matched and put it into $1, $2, and $3 - special variables called backreferences. As you can see, I use these backreferences in the replacement string.

Task #3. Number the files with a 2 digit counter, starting from 00.

Rename mask: file name: [C]. [N]
Define counter (C) Start at: 0
Define counter (C) Step by: 1
Define counter (C) Digits: 2

In this case, I've used a special [C] variable (this is unrelated to regex) that stands for counter and customized the counter setup. You can see some other useful variables in the screenshot.

Task #4. Rename 4 letter extensions into 3 letter ones (Excel 2007 xlsx -> Excel 2003 xls).

Let's say  now that my files have a .xlsx extension that was introduced in Excel 2007. There are actually a few ways I can think of renaming them in Total Commander, the easiest being selecting all files, pressing Shift-F6 (inline rename) and then entering *.xls. BAM! All files are now .xls. The problem is that this wouldn't work if I had a mix of .docx and .xlsx files, for instance. Another solution would be to use regex, but I want to try something different. Here goes:

Extension: [E1-3]

This approach replaces file extensions with the extension sub-string [1-3].

There are plenty of other tasks Total Commander lets me accomplish very efficiently, like dealing with archives, files in subdirectories, checksums, copying lists of files to clipboard, comparing directories, FTPing, searching, etc, etc, etc. Maybe I'll write about them some day. So go out there, download it, and I guarantee it will change the way you think of file management forever.

Appendix A. Total Commander Multi-Rename Tool manual.

With this dialog box, you can rename a list of files selected in Total Commander. Instead of * and ? wildcards, this function uses placeholders in brackets []. The new names are immediately shown in the result list, but the files are not renamed until the Start! button is pressed.

Field    Description

Rename mask: file name
    With this field, you can create a definition for a new file name. The buttons below allow to insert place holders for the previous name, parts of the name, a counter, or file date/time. Place holders are always in brackets [ ], while all other letters (without brackets) will be placed in the new name without a change.
    See below for a description of all available placeholders! Use Shift+Del to remove no longer wanted entries from the list.

Extension    Definition string for extension. In principle, all placeholders can be added to either of the two definition boxes. The rename tool will create a rename string like this: Fields in name mask + "." + Fields in extension mask. The reason why the two fields are separated is to prevent the accidental removal of file extensions, which would remove the association of files with a certain program.

Search & Replace    The string entered in the field 'Search for' is replaced by the string in 'Replace with'. The text in 'search for' is NOT case sensitive! Both fields support the placeholders (wildcards) * and ?. A * stands for any number of characters, a ? for exactly one character.

This function is applied AFTER the rename mask!
    New: You can now search+replace multiple strings in one step! The strings need to be separated by the vertical line (Alt+124).
    Example: Replace Umlauts+Accents:
    Search for: ä|ö|ü|é|è|ê|à  Replace with: ae|oe|ue|e|e|e|a

^    Respect upper-/lowercase. Can be used to replace uppercase characters with other charters than lowercase.
    Example: Replace Umlauts+Accents considering upper/lowercase:

Search for: ä|ö|ü|é|è|ê|à|Ä|Ö|Ü|É|È|Ê|À  Replace with: ae|oe|ue|e|e|e|a|AE|OE|UE|E|E|E|A

RegEx    Now supports regular expressions.

Subst.    Substitution: The entire file name will be replaced by the characters entered in the "Replace" field. If this option isn't checked, only the found expression will be replaced. You can work with subexpressions, see the sample in regular expressions.

Upper/lowercase    Converts the whole string to uppercase/lowercase/first letter uppercase,rest lowercase. This function is applied AFTER the rename mask and after search&replace. Use the [U], [L] and [n] placeholders to convert only certain parts of the name to upper/lowercase!

    Opens a context menu with the following options:
    Load names from file: Specify a text file from which the new names should be loaded
    Edit names: Save current target names to a text file, edit them, and paste back.

Configure editor: Choose a different editor to edit the files. Make sure that the editor saves as plain text!

Define counter [C]    Allows to define the counter for the [C] field(s).
Start at:    Number of the first file. The files are always numbered the same way as they are shown in the result list. You can sort the result list just like in the main Total Commander window. Additionally you can reorder individual items using drag&drop, or Shift+Cursor keys.

Step by:    The counter is increased/decreased by this value.
Digits:    Width of the counter field. If digits is >1, the rename tool will insert leading zeros to get a fixed width number field.

F2 Load/save settings
    Allows to load or save the settings of the multi-rename tool.
<Default>    Sets the default settings (no changes to the names)
Save settings
    Allows to save the current settings
Delete entry    Deletes the last selected entry from the list

Entry names    Loads settings saved earlier

<File list header>    Allows to sort by old names, extensions, sizes or time stamps
<File list>    Shows a list of all files being renamed. The modified names are shown in the New name column. All changes to the above fields are immediately shown in this column, but the files are not actually renamed until the Start! button is pressed. If the rename mask contains an error, the string <Error!> is shown.

Individual items can be moved up/down with drag&drop or Shift+Cursor keys. This is useful for changing the file order for the counter function.

(next step)    Loads the rename results for the next rename step. Useful if you want to apply multiple rename rules to the same set of files. Shortcut: F5.
Start!    Starts to rename files. There will be a warning message if there are name conflicts.
Undo    Tries to undo the rename operation in reverse order (last renamed file first). This also works AFTER closing the Multi-Rename-Tool! Just re-open it with any file(s).

Result list    Creates a protocol of the renamed files.
Close    Closes the dialog box without any further actions.

Here is a description of all available placeholders. IMPORTANT: Upper/lowercase is relevant!
[N]    old file name, WITHOUT extension
[N2-5]    Characters 2 to 5 from the old name (totals to 4 characters). Double byte characters (e.g. Chinese, Japanese) are counted as 1 character! The first letter is accessed with '1'.
[N2,5]    5 characters starting at character 2
[N2-]    All characters starting at character 2
[N-8,5]    5 characters starting at the 8-last character (counted from the end of the name)

[N-8-5]    Characters from the 8th-last to the 5th-last character
[N2--5]    Characters from the 2nd to the 5th-last character
[N-5-]    Characters from the 5th-last character to the end of the name
[2-5]    Characters 2-5 from the whole name and extension (other numbers as in [N] definition)
[P]    Paste name of the parent directory, e.g. when renaming c:\directory\file.txt -> pastes "directory".
    Also working: [P2-5], [P2,5], [P-8,5], [P-8-5] and [P2-], see description of [N] above.

[G]    Grandparent directory (usage: see [P]).
[E]    Extension
[E1-2]    Characters 1-2 from the extension (other numbers as in [N] definition)
[C]    Paste counter, as defined in Define counter field
[C10+5:3]    Paste counter, define counter settings directly. In this example, start at 10, step by 5, use 3 digits width.
    Partial definitions like [C10] or [C+5] or [C:3] are also accepted.
[Caa+1]    Paste counter, define counter settings directly. In this example, start at aa, step 1 letter, use 2 digits (defined by 'aa' width)

[d]    Paste date as defined in current country settings. / is replaced by a dash
[Y]    Paste year in 4 digit form
[y]    Paste year in 2 digit form
[M]    Paste month, always 2 digit
[D]    Paste day, always 2 digit
[t]    Paste time, as defined in current country settings. : is replaced by a dot.
[h]    Paste hours, always in 24 hour 2 digit format
[m]    Paste minutes, always in 2 digit format
[s]    Paste seconds, always in 2 digit format

[U]    All characters after this position in uppercase
[L]    All characters after this position in lowercase
[F]    First letter of each word uppercase after this position, all others lowercase
[n]    All characters after this position again as in original name (upper/lowercase unchanged)
[[]    Insert square bracket: open
[]]    Insert square bracket: close (cannot be combined with other commands inside the square bracket!)
[=pluginname.fieldname.unit]

Insert field named "fieldname" from content plugin named "pluginname". "unit" may be an optional unit (if supported by that field), or a field formatter like YMD for date fields. You can use the [=?] Plugin button to insert plugin fields.
[=pluginname.fieldname.unit:4-7]
    Same as above, but for partial strings (here: letters 4-7).

Appendix B. Total Commander regex support.

Regular expressions are a very powerful search tool. They allow to search for complex classes of words. Regular expressions are mainly meant for professionals, but can also be useful in the office for finding certain documents (see examples below).

Total Commander supports regular expressions in the following functions:
- Commands - Search (in file name and file contents)
- In Lister
- In the Multi-Rename tool
- In the selection dialog

Regular expressions consist of normal characters and special characters, so-called meta-characters. The following characters are meta-characters or initial parts of meta-characters:
.  \  (  )  [  ]  {  }  ^  $  +  *  ?    (only in character classes: - )

Normal characters:

test    finds the string "test" in the searched text. Note: This finds "test" ANYWHERE in a file name or on a line in text.

Escape sequences:

A backslash \ starts an Escape sequence. Examples for escape sequences:

\t    Tabstop
\xnn    Character with hexadecimal code nn. Example: \x20 is the space character. The character table charmap.exe (if installed) shows the character code of most special characters. You can use the Windows calculator in scientific mode to convert from decimal to hex.
\[    Left square bracket. Since the square brackets are meta-characters, they need to be written as \[ to search for them in the target string.
\\    Finds a backslash.
\.    Finds a dot ("." alone finds any character, see below).

Character classes

Characters in square brackets build a character class. It will find exacly one character from this class. A dash allows to define groups, e.g. [a-z]. A ^ at the beginning finds all characters except for those listed.
Examples:

[aeiou]    Finds exactly one of the listed vovels.
[^aeiou]    Finds everything except for a vovel.
M[ae][iy]er    Finds a Mr. Meier in all possible ways of writing: Mayer, Meyer, Maier, Meier. Very useful if you cannot remember the exact writing of a name.

Meta-characters

Here is a list of the most important meta-characters:

^    Line start
$    Line end
.    Any character
\w    a letter, digit or underscore _
\W    the opposite of \w
\d    a digit
\D    no digit
\s    a word separator (space, tab etc)
\S    no word separator
\b    finds a word boundary (combination of \s and \S)
\B    the opposite of \b

Iterators

Iterators are used for a repetition of the character or expression to the left of the iterator.
*    zero or more occurances
+    one or more occurances
{n}    exactly n occurances
{n,}    at least n occurances

{n,m}    at least n and max. m occurances

All these operators are "greedy", which means that they take as many characters as they can get. Putting a question mark ? after an operator makes it "non-greedy", i.e. it takes only as many characters as needed.
Example: "b+" applied to the target string "abbbbc" finds "bbbb", "b+?" finds just "b".

Alternatives

Alternatives are put in round braces, and are separated by a vertical dash.

Example: (John|James|Peter)  finds one of the names John, James or Peter.

Subexpressions for search+replace

Text parts in round braces are taken as subexpressions.
Example: To swap the title and interpret in the file name of an mp3 file, when they are separated by a dash (Title - Interpret.mp3), this can be solved like this:
Search for: (.*) - (.*)\.mp3
Replace by: $2 - $1.mp3
Here $1 means the text in the first brace, and $2 the text in the second brace.

Backreferences

\n    Finds subexpression n another time in the search result.

Example: (.+)\1+  finds e.g.  abab  (where the first ab is found by .+  and the second by \1+ )

Modifiers

Modifiers are used for changing behaviour of regular expressions.

(?i)    Ignore Upper-/lowercase. In Total Commander, this is the default for file names.
(?-i)    Case-sensitive matching.
(?g)    Switches on "greedy" mode (active by default)
(?-g)    Turns off "greedy" mode, so "+" means the same as "+?"

The other modificators are not relevant for Total Commander, because the program only supports searching within one line.

Total Commander uses the free Delphi library TRegExpr by Andrey V. Sorokin: http://www.regexpstudio.com/
Some of the above explanations are from the help file for this library.

Updated: March 18th, 2008

"Didn't see that one coming. Their blog contains details to what this could mean for both companies. May as well be one of the most important takeovers of 2008 already!"

read more | digg story

Could this mean that the mysql cluster is finally going to get proper development attention? I don't know but sure as hell hope so. Congratulations to all mysql employees!